[PATCH iproute2 V3 0/4] RDMAtool

[PATCH iproute2 V3 0/4] RDMAtool

[Leon Romanovsky]

Hi,

This is third version of series implementing the RDAMtool -  the tool
to configure RDMA devices. The initial proposal was sent as RFC [1] and
was based on sysfs entries as POC.

The current series was rewritten completely to work with RDMA netlinks as
a source of user<->kernel communications. In order to achieve that, the
RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].

The following is an example of various runs on my machine with 5 devices
(4 in IB mode and one in Ethernet mode)

### Without parameters
$ rdma
Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
where  OBJECT := { dev | link | help }
       OPTIONS := { -V[ersion] | -d[etails]}

### With unspecified device name
$ rdma dev
1: mlx5_0: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3457 sys_image_guid 5254:00c0:fe12:3457
2: mlx5_1: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3458 sys_image_guid 5254:00c0:fe12:3458
3: mlx5_2: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3459 sys_image_guid 5254:00c0:fe12:3459
4: mlx5_3: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345a sys_image_guid 5254:00c0:fe12:345a
5: mlx5_4: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345b sys_image_guid 5254:00c0:fe12:345b

### Detailed mode
$ rdma -d dev
1: mlx5_0: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3457 sys_image_guid 5254:00c0:fe12:3457
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>
2: mlx5_1: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3458 sys_image_guid 5254:00c0:fe12:3458
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>
3: mlx5_2: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:3459 sys_image_guid 5254:00c0:fe12:3459
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>
4: mlx5_3: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345a sys_image_guid 5254:00c0:fe12:345a
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>
5: mlx5_4: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345b sys_image_guid 5254:00c0:fe12:345b
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>

### Specific device
$ rdma dev show mlx5_4
5: mlx5_4: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345b sys_image_guid 5254:00c0:fe12:345b

### Specific device in detailed mode
$ rdma dev show mlx5_4 -d
5: mlx5_4: node_type SWITCH fw 2.8.9999 node_guid 5254:00c0:fe12:345b sys_image_guid 5254:00c0:fe12:345b
    caps: <BAD_PKEY_CNTR, BAD_QKEY_CNTR, CHANGE_PHY_PORT, PORT_ACTIVE_EVENT, SYS_IMAGE_GUID, RC_RNR_NAK_GEN, MEM_WINDOW, UD_IP_CSUM, UD_TSO, XRC, MEM_MGT_EXTENSIONS, BLOCK_MULTICAST_LOOPBACK, MEM_WINDOW_TYPE_2B, RAW_IP_CSUM, SIGNATURE_HANDOVER, VIRTUAL_FUNCTION>

### Unknown command (caps)
$ rdma dev show mlx5_4 caps
Unknown parameter 'caps'.

### Link properties without device name
$ rdma link
1/1: mlx5_0/1: subnet_prefix fe80:0000:0000:0000 lid 13399 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
2/1: mlx5_1/1: subnet_prefix fe80:0000:0000:0000 lid 13400 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
3/1: mlx5_2/1: subnet_prefix fe80:0000:0000:0000 lid 13401 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
4/1: mlx5_3/1: state DOWN physical_state DISABLED
5/1: mlx5_4/1: subnet_prefix fe80:0000:0000:0000 lid 13403 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP

### Link properties in detailed mode
$ rdma link -d
1/1: mlx5_0/1: subnet_prefix fe80:0000:0000:0000 lid 13399 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
    caps: <AUTO_MIGR>
2/1: mlx5_1/1: subnet_prefix fe80:0000:0000:0000 lid 13400 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
    caps: <AUTO_MIGR>
3/1: mlx5_2/1: subnet_prefix fe80:0000:0000:0000 lid 13401 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
    caps: <AUTO_MIGR>
4/1: mlx5_3/1: state DOWN physical_state DISABLED
    caps: <CM, IP_BASED_GIDS>
5/1: mlx5_4/1: subnet_prefix fe80:0000:0000:0000 lid 13403 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
    caps: <AUTO_MIGR>

### All links for specific device
$ rdma link show mlx5_3
1/1: mlx5_0/1: subnet_prefix fe80:0000:0000:0000 lid 13399 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP

### Detailed link properties for specific device
$ rdma link -d show mlx5_3
1/1: mlx5_0/1: subnet_prefix fe80:0000:0000:0000 lid 13399 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP
    caps: <AUTO_MIGR>

### Specific port for specific device
$ rdma link show mlx5_4/1
1/1: mlx5_0/1: subnet_prefix fe80:0000:0000:0000 lid 13399 sm_lid 49151 lmc 0 state ACTIVE physical_state LINK_UP

### Unknown parameter
$ rdma link show mlx5_4/1 caps
Unknown parameter 'caps'.

Thanks

Changelog:
v2->v3:
 * Removed MAX()
 * Reduced scope of rd_argv_match
 * Removed return from rdma_free_devmap
 * Added extra break at rdma_send_msg
v1->v2:
 * Squashed multiple (and similar) patches to be one patch for dev object
   and one patch for link object.
 * Removed port_map struct
 * Removed global netlink dump during initialization, it removed the need to store
   the intermediate variables and reuse ability of netlink to signal if variable
   exists or doesn't.
 * Added "-d" --details option and put all CAPs under it.

v0->v1:
 * Moved hunk with changes in man/Makefile from first patch to the last patch
 * Removed the "unknown command" from the examples in commit messages
 * Removed special "caps" parsing command and put it to be part of general "show" command
 * Changed parsed capability format to be similar to iproute2 suite
 * Added FW version as an output of show command.
 * Added forgotten CAP_FLAGS to the nla_policy list
RFC->v0:
 * Removed everything that is not implemented yet.
 * Abandoned sysfs interfaces in favor of netlink.

Available in the "topic/rdmatool-netlink-v2" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/iproute2.git/log/?h=topic/rdmatool-netlink-v2

Thanks

[1] https://www.spinics.net/lists/linux-rdma/msg49575.html
[2] https://patchwork.kernel.org/patch/9752865/
[3] https://www.spinics.net/lists/linux-rdma/msg50827.html
[4] https://www.spinics.net/lists/linux-rdma/msg51210.html
[5] https://patchwork.kernel.org/patch/9811729/ and https://patchwork.kernel.org/patch/9811731/]

Cc: Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Ariel Almog <ariela-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Linux RDMA <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Cc: Linux Netdev <netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>

Leon Romanovsky (4):
  rdma: Add basic infrastructure for RDMA tool
  rdma: Add dev object
  rdma: Add link object
  rdma: Add initial manual for the tool

 Makefile          |   2 +-
 man/man8/Makefile |   3 +-
 man/man8/rdma.8   |  82 ++++++++++++++++
 rdma/.gitignore   |   1 +
 rdma/Makefile     |  22 +++++
 rdma/dev.c        | 235 +++++++++++++++++++++++++++++++++++++++++++++
 rdma/link.c       | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.c       | 118 +++++++++++++++++++++++
 rdma/rdma.h       |  80 ++++++++++++++++
 rdma/utils.c      | 270 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 10 files changed, 1091 insertions(+), 2 deletions(-)
 create mode 100644 man/man8/rdma.8
 create mode 100644 rdma/.gitignore
 create mode 100644 rdma/Makefile
 create mode 100644 rdma/dev.c
 create mode 100644 rdma/link.c
 create mode 100644 rdma/rdma.c
 create mode 100644 rdma/rdma.h
 create mode 100644 rdma/utils.c

--
2.13.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH iproute2 V3 1/4] rdma: Add basic infrastructure for RDMA tool

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

RDMA devices are cross-functional devices from one side,
but very tailored for the specific markets from another.

Such diversity caused to spread of RDMA related configuration
across various tools, e.g. devlink, ip, ethtool, ib specific and
vendor specific solutions.

This patch adds ability to fill device and port information
by reading RDMA netlink.

Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 Makefile        |   2 +-
 rdma/.gitignore |   1 +
 rdma/Makefile   |  22 ++++++
 rdma/rdma.c     | 116 +++++++++++++++++++++++++++++
 rdma/rdma.h     |  70 ++++++++++++++++++
 rdma/utils.c    | 223 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 433 insertions(+), 1 deletion(-)
 create mode 100644 rdma/.gitignore
 create mode 100644 rdma/Makefile
 create mode 100644 rdma/rdma.c
 create mode 100644 rdma/rdma.h
 create mode 100644 rdma/utils.c

diff --git a/Makefile b/Makefile
index 18de7dcb..c255063b 100644
--- a/Makefile
+++ b/Makefile
@@ -52,7 +52,7 @@ WFLAGS += -Wmissing-declarations -Wold-style-definition -Wformat=2
 CFLAGS := $(WFLAGS) $(CCOPTS) -I../include $(DEFINES) $(CFLAGS)
 YACCFLAGS = -d -t -v

-SUBDIRS=lib ip tc bridge misc netem genl tipc devlink man
+SUBDIRS=lib ip tc bridge misc netem genl tipc devlink rdma man

 LIBNETLINK=../lib/libnetlink.a ../lib/libutil.a
 LDLIBS += $(LIBNETLINK)
diff --git a/rdma/.gitignore b/rdma/.gitignore
new file mode 100644
index 00000000..51fb172b
--- /dev/null
+++ b/rdma/.gitignore
@@ -0,0 +1 @@
+rdma
diff --git a/rdma/Makefile b/rdma/Makefile
new file mode 100644
index 00000000..64da2142
--- /dev/null
+++ b/rdma/Makefile
@@ -0,0 +1,22 @@
+include ../Config
+
+ifeq ($(HAVE_MNL),y)
+
+RDMA_OBJ = rdma.o utils.o
+
+TARGETS=rdma
+CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
+LDLIBS += $(shell $(PKG_CONFIG) libmnl --libs)
+
+endif
+
+all:	$(TARGETS) $(LIBS)
+
+rdma:	$(RDMA_OBJ) $(LIBS)
+	$(QUIET_LINK)$(CC) $^ $(LDFLAGS) $(LDLIBS) -o $@
+
+install: all
+	install -m 0755 $(TARGETS) $(DESTDIR)$(SBINDIR)
+
+clean:
+	rm -f $(RDMA_OBJ) $(TARGETS)
diff --git a/rdma/rdma.c b/rdma/rdma.c
new file mode 100644
index 00000000..29273839
--- /dev/null
+++ b/rdma/rdma.c
@@ -0,0 +1,116 @@
+/*
+ * rdma.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include <limits.h>
+#include <rdma/rdma_netlink.h>
+
+#include "rdma.h"
+#include "SNAPSHOT.h"
+
+static void help(char *name)
+{
+	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
+	       "where  OBJECT := { help }\n"
+	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
+}
+
+static int cmd_help(struct rdma *rd)
+{
+	help(rd->filename);
+	return 0;
+}
+
+static int rd_cmd(struct rdma *rd)
+{
+	const struct rdma_cmd cmds[] = {
+		{ NULL,		cmd_help },
+		{ "help",	cmd_help },
+		{ 0 }
+	};
+
+	return rdma_exec_cmd(rd, cmds, "object");
+}
+
+static int rd_init(struct rdma *rd, int argc, char **argv, char *filename)
+{
+	uint32_t seq;
+	int ret;
+
+	rd->filename = filename;
+	rd->argc = argc;
+	rd->argv = argv;
+	INIT_LIST_HEAD(&rd->dev_map_list);
+	rd->buff = malloc(MNL_SOCKET_BUFFER_SIZE);
+	if (!rd->buff)
+		return -ENOMEM;
+
+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK | NLM_F_DUMP));
+	if ((ret = rdma_send_msg(rd)))
+		return ret;
+
+	return rdma_recv_msg(rd, rd_dev_init_cb, rd, seq);
+}
+
+static void rd_free(struct rdma *rd)
+{
+	free(rd->buff);
+	rdma_free_devmap(rd);
+}
+int main(int argc, char **argv)
+{
+	static const struct option long_options[] = {
+		{ "version",		no_argument,		NULL, 'V' },
+		{ "help",		no_argument,		NULL, 'h' },
+		{ "details",		no_argument,		NULL, 'd' },
+		{ NULL, 0, NULL, 0 }
+	};
+	bool show_details = false;
+	char *filename;
+	struct rdma rd;
+	int opt;
+	int err;
+
+	filename = basename(argv[0]);
+
+	while ((opt = getopt_long(argc, argv, "Vhd",
+				  long_options, NULL)) >= 0) {
+
+		switch (opt) {
+		case 'V':
+			printf("%s utility, iproute2-ss%s\n", filename, SNAPSHOT);
+			return EXIT_SUCCESS;
+		case 'd':
+			show_details = true;
+			break;
+		case 'h':
+			help(filename);
+			return EXIT_SUCCESS;
+		default:
+			pr_err("Unknown option.\n");
+			help(filename);
+			return EXIT_FAILURE;
+		}
+	}
+
+	argc -= optind;
+	argv += optind;
+
+	err = rd_init(&rd, argc, argv, filename);
+	if (err)
+		goto out;
+
+	rd.show_details = show_details;
+	err = rd_cmd(&rd);
+out:
+	/* Always cleanup */
+	rd_free(&rd);
+	return (err) ? EXIT_FAILURE:EXIT_SUCCESS;
+}
diff --git a/rdma/rdma.h b/rdma/rdma.h
new file mode 100644
index 00000000..0c01b091
--- /dev/null
+++ b/rdma/rdma.h
@@ -0,0 +1,70 @@
+/*
+ * rdma.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+#ifndef _RDMA_TOOL_H_
+#define _RDMA_TOOL_H_
+
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <getopt.h>
+#include <libmnl/libmnl.h>
+
+#include <rdma/rdma_netlink.h>
+#include "list.h"
+
+#define pr_err(args...) fprintf(stderr, ##args)
+#define pr_out(args...) fprintf(stdout, ##args)
+
+struct dev_map {
+	struct list_head list;
+	char *dev_name;
+	uint32_t num_ports;
+	uint32_t idx;
+};
+
+struct rdma {
+	int argc;
+	char **argv;
+	char *filename;
+	bool show_details;
+	struct list_head dev_map_list;
+	struct mnl_socket *nl;
+	struct nlmsghdr *nlh;
+	char *buff;
+};
+
+struct rdma_cmd {
+	const char *cmd;
+	int (*func)(struct rdma *rd);
+};
+
+/*
+ * Parser interface
+ */
+bool rd_no_arg(struct rdma *rd);
+void rd_arg_inc(struct rdma *rd);
+
+int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *c, const char *str);
+
+/*
+ * Device manipulation
+ */
+void rdma_free_devmap(struct rdma *rd);
+
+/*
+ * Netlink
+ */
+int rdma_send_msg(struct rdma *rd);
+int rdma_recv_msg(struct rdma *rd, mnl_cb_t callback, void *data, uint32_t seq);
+void rdma_prepare_msg(struct rdma *rd, uint32_t cmd, uint32_t *seq, uint16_t flags);
+int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data);
+int rd_attr_cb(const struct nlattr *attr, void *data);
+#endif /* _RDMA_TOOL_H_ */
diff --git a/rdma/utils.c b/rdma/utils.c
new file mode 100644
index 00000000..b554eff7
--- /dev/null
+++ b/rdma/utils.c
@@ -0,0 +1,223 @@
+/*
+ * utils.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include <sys/types.h>
+#include <dirent.h>
+#include <time.h>
+
+#include "rdma.h"
+#include <rdma/rdma_netlink.h>
+
+static int rd_argc(struct rdma *rd)
+{
+	return rd->argc;
+}
+
+static char *rd_argv(struct rdma *rd)
+{
+	if (!rd_argc(rd))
+		return NULL;
+	return *rd->argv;
+}
+
+static int strcmpx(const char *str1, const char *str2)
+{
+	if (strlen(str1) > strlen(str2))
+			return -1;
+	return strncmp(str1, str2, strlen(str1));
+}
+
+static bool rd_argv_match(struct rdma *rd, const char *pattern)
+{
+	if (!rd_argc(rd))
+		return false;
+	return strcmpx(rd_argv(rd), pattern) == 0;
+}
+
+void rd_arg_inc(struct rdma *rd)
+{
+	if (!rd_argc(rd))
+		return;
+	rd->argc--;
+	rd->argv++;
+}
+
+bool rd_no_arg(struct rdma *rd)
+{
+	return rd_argc(rd) == 0;
+}
+
+static struct dev_map *dev_map_alloc(const char *dev_name)
+{
+	struct dev_map *dev_map;
+
+	dev_map = calloc(1, sizeof(*dev_map));
+	if (!dev_map)
+		return NULL;
+	dev_map->dev_name = strdup(dev_name);
+
+	return dev_map;
+}
+
+static void dev_map_free(struct dev_map *dev_map)
+{
+	if(!dev_map)
+		return;
+
+	free(dev_map->dev_name);
+	free(dev_map);
+}
+
+static void dev_map_cleanup(struct rdma *rd)
+{
+	struct dev_map *dev_map, *tmp;
+
+	list_for_each_entry_safe(dev_map, tmp,
+				 &rd->dev_map_list, list) {
+		list_del(&dev_map->list);
+		dev_map_free(dev_map);
+	}
+}
+
+static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
+	[RDMA_NLDEV_ATTR_DEV_NAME] = MNL_TYPE_NUL_STRING,
+	[RDMA_NLDEV_ATTR_PORT_INDEX] = MNL_TYPE_U32,
+};
+
+int rd_attr_cb(const struct nlattr *attr, void *data)
+{
+	const struct nlattr **tb = data;
+	int type;
+
+	if (mnl_attr_type_valid(attr, RDMA_NLDEV_ATTR_MAX) < 0)
+		return MNL_CB_ERROR;
+
+	type = mnl_attr_get_type(attr);
+
+	if (mnl_attr_validate(attr, nldev_policy[type]) < 0)
+		return MNL_CB_ERROR;
+
+	tb[type] = attr;
+	return MNL_CB_OK;
+}
+
+int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)
+{
+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
+	struct dev_map *dev_map;
+	struct rdma *rd = data;
+	const char *dev_name;
+
+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
+	if (!tb[RDMA_NLDEV_ATTR_DEV_NAME] || !tb[RDMA_NLDEV_ATTR_DEV_INDEX])
+		return MNL_CB_ERROR;
+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
+		pr_err("This tool doesn't support switches yet\n");
+		return MNL_CB_ERROR;
+	}
+
+	dev_name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
+
+	dev_map = dev_map_alloc(dev_name);
+	if (!dev_map)
+		/* The main function will cleanup the allocations */
+		return MNL_CB_ERROR;
+	list_add_tail(&dev_map->list, &rd->dev_map_list);
+
+	dev_map->num_ports = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
+	dev_map->idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
+
+	return MNL_CB_OK;
+}
+
+void rdma_free_devmap(struct rdma *rd)
+{
+	if(!rd)
+		return;
+	dev_map_cleanup(rd);
+}
+
+int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *cmds, const char *str)
+{
+	const struct rdma_cmd *c;
+
+	/* First argument in objs table is default variant */
+	if (rd_no_arg(rd))
+		return cmds->func(rd);
+
+	for (c = cmds + 1; c->cmd; ++c) {
+		if (rd_argv_match(rd, c->cmd)) {
+			/* Move to next argument */
+			rd_arg_inc(rd);
+			return c->func(rd);
+		}
+	}
+
+	pr_err("Unknown %s '%s'.\n", str, rd_argv(rd));
+	return 0;
+}
+
+void rdma_prepare_msg(struct rdma *rd, uint32_t cmd, uint32_t *seq, uint16_t flags)
+{
+	*seq = time(NULL);
+
+	rd->nlh = mnl_nlmsg_put_header(rd->buff);
+	rd->nlh->nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, cmd);
+	rd->nlh->nlmsg_seq = *seq;
+	rd->nlh->nlmsg_flags = flags;
+}
+
+int rdma_send_msg(struct rdma *rd)
+{
+	int ret;
+
+	rd->nl = mnl_socket_open(NETLINK_RDMA);
+	if (!rd->nl) {
+		pr_err("Failed to open NETLINK_RDMA socket\n");
+		return -ENODEV;
+	}
+
+	ret = mnl_socket_bind(rd->nl, 0, MNL_SOCKET_AUTOPID);
+	if (ret < 0) {
+		pr_err("Failed to bind socket with err %d\n", ret);
+		goto err;
+	}
+
+	ret = mnl_socket_sendto(rd->nl, rd->nlh, rd->nlh->nlmsg_len);
+	if (ret < 0) {
+		pr_err("Failed to send to socket with err %d\n", ret);
+		goto err;
+	}
+	return 0;
+
+err:
+	mnl_socket_close(rd->nl);
+	return ret;
+}
+
+int rdma_recv_msg(struct rdma *rd, mnl_cb_t callback, void *data, unsigned int seq)
+{
+	int ret;
+	unsigned int portid;
+	char buf[MNL_SOCKET_BUFFER_SIZE];
+
+	portid = mnl_socket_get_portid(rd->nl);
+	do {
+		ret = mnl_socket_recvfrom(rd->nl, buf, sizeof(buf));
+		if (ret <= 0)
+			break;
+
+		ret = mnl_cb_run(buf, ret, seq, portid, callback, data);
+	} while (ret > 0);
+
+	mnl_socket_close(rd->nl);
+	return ret;
+}

[PATCH iproute2 V3 2/4] rdma: Add dev object

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

Device (dev) object represents struct ib_device to the user space.

Device properties:
 * Device capabilities
 * FW version to the device output
 * node_guid and sys_image_guid
 * node_type

Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 rdma/Makefile |   2 +-
 rdma/dev.c    | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.c   |   3 +-
 rdma/rdma.h   |  12 ++-
 rdma/utils.c  |  46 +++++++++++-
 5 files changed, 293 insertions(+), 5 deletions(-)
 create mode 100644 rdma/dev.c

diff --git a/rdma/Makefile b/rdma/Makefile
index 64da2142..123d7ac5 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile
@@ -2,7 +2,7 @@ include ../Config

 ifeq ($(HAVE_MNL),y)

-RDMA_OBJ = rdma.o utils.o
+RDMA_OBJ = rdma.o utils.o dev.o

 TARGETS=rdma
 CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
diff --git a/rdma/dev.c b/rdma/dev.c
new file mode 100644
index 00000000..b80e5288
--- /dev/null
+++ b/rdma/dev.c
@@ -0,0 +1,235 @@
+/*
+ * dev.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static int dev_help(struct rdma *rd)
+{
+	pr_out("Usage: %s dev show [DEV]\n", rd->filename);
+	return 0;
+}
+
+static void dev_print_caps(struct nlattr **tb)
+{
+	uint64_t caps;
+	uint32_t idx;
+
+	/*
+	 * FIXME: move to indexes when kernel will start exporting them.
+	 */
+	static const char *dev_caps[64] = {
+		"RESIZE_MAX_WR",
+		"BAD_PKEY_CNTR",
+		"BAD_QKEY_CNTR",
+		"RAW_MULTI",
+		"AUTO_PATH_MIG",
+		"CHANGE_PHY_PORT",
+		"UD_AV_PORT_ENFORCE",
+		"CURR_QP_STATE_MOD",
+		"SHUTDOWN_PORT",
+		"INIT_TYPE",
+		"PORT_ACTIVE_EVENT",
+		"SYS_IMAGE_GUID",
+		"RC_RNR_NAK_GEN",
+		"SRQ_RESIZE",
+		"N_NOTIFY_CQ",
+		"LOCAL_DMA_LKEY",
+		"RESERVED",
+		"MEM_WINDOW",
+		"UD_IP_CSUM",
+		"UD_TSO",
+		"XRC",
+		"MEM_MGT_EXTENSIONS",
+		"BLOCK_MULTICAST_LOOPBACK",
+		"MEM_WINDOW_TYPE_2A",
+		"MEM_WINDOW_TYPE_2B",
+		"RC_IP_CSUM",
+		"RAW_IP_CSUM",
+		"CROSS_CHANNEL",
+		"MANAGED_FLOW_STEERING",
+		"SIGNATURE_HANDOVER",
+		"ON_DEMAND_PAGING",
+		"SG_GAPS_REG",
+		"VIRTUAL_FUNCTION",
+		"RAW_SCATTER_FCS",
+		"RDMA_NETDEV_OPA_VNIC",
+	};
+
+	if (!tb[RDMA_NLDEV_ATTR_CAP_FLAGS])
+	       return;
+
+	caps = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_CAP_FLAGS]);
+
+	pr_out("\n    caps: <");
+	for (idx = 0; idx < 64; idx++) {
+		if (caps & 0x1) {
+			pr_out("%s", dev_caps[idx]?dev_caps[idx]:"UNKNONW");
+			if (caps >> 0x1)
+				pr_out(", ");
+		}
+		caps >>= 0x1;
+	}
+
+	pr_out(">");
+}
+
+static void dev_print_fw(struct nlattr **tb)
+{
+	if (!tb[RDMA_NLDEV_ATTR_FW_VERSION])
+		return;
+
+	pr_out("fw %s ",
+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_FW_VERSION]));
+}
+
+static void _dev_print_be64(char *name, uint64_t val)
+{
+	uint16_t vp[4];
+
+	memcpy(vp, &val, sizeof(uint64_t));
+	pr_out("%s %04x:%04x:%04x:%04x ", name, vp[3], vp[2], vp[1], vp[0]);
+}
+
+static void dev_print_node_guid(struct nlattr **tb)
+{
+	uint64_t node_guid;
+
+	if (!tb[RDMA_NLDEV_ATTR_NODE_GUID])
+		return;
+
+	node_guid = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_NODE_GUID]);
+	_dev_print_be64("node_guid", node_guid);
+}
+
+static void dev_print_sys_image_guid(struct nlattr **tb)
+{
+       uint64_t	sys_image_guid;
+
+	if (!tb[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID])
+		return;
+
+	sys_image_guid = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID]);
+	_dev_print_be64("sys_image_guid", sys_image_guid);
+}
+
+static void dev_print_node_type(struct nlattr **tb)
+{
+	uint8_t node_type;
+	/*
+	 * FIXME: move to index exported by the kernel
+	 */
+	static const char *str[] = {
+		"UNKNOWN",
+		"SWITCH",
+		"ROUTER",
+		"RNIC",
+		"USNIC",
+		"USNIC_UDP",
+	};
+
+	if (!tb[RDMA_NLDEV_ATTR_DEV_NODE_TYPE])
+		return;
+
+	node_type = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_DEV_NODE_TYPE]);
+
+	if (node_type < 7 )
+		pr_out("node_type %s ", str[node_type]);
+	else
+		pr_out("node_type UNKNOWN ");
+}
+
+static int dev_parse_cb(const struct nlmsghdr *nlh, void *data)
+{
+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
+	struct rdma *rd = data;
+
+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
+	if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_DEV_NAME])
+		return MNL_CB_ERROR;
+
+	pr_out("%u: %s: ",
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]),
+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]));
+	dev_print_node_type(tb);
+	dev_print_fw(tb);
+	dev_print_node_guid(tb);
+	dev_print_sys_image_guid(tb);
+	if (rd->show_details)
+		dev_print_caps(tb);
+
+	pr_out("\n");
+	return MNL_CB_OK;
+}
+
+
+static int dev_no_args(struct rdma *rd)
+{
+	uint32_t seq;
+	int ret;
+
+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK));
+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
+	if ((ret = rdma_send_msg(rd)))
+		return ret;
+
+	return rdma_recv_msg(rd, dev_parse_cb, rd, seq);
+}
+
+static int dev_one_show(struct rdma *rd)
+{
+	const struct rdma_cmd cmds[] = {
+		{ NULL,		dev_no_args},
+		{ 0 }
+	};
+
+	return rdma_exec_cmd(rd, cmds, "parameter");
+
+}
+
+static int dev_show(struct rdma *rd)
+{
+	struct dev_map *dev_map;
+	int ret = 0;
+
+	if (rd_no_arg(rd)) {
+		list_for_each_entry(dev_map, &rd->dev_map_list, list) {
+			rd->dev_idx = dev_map->idx;
+			ret = dev_one_show(rd);
+			if (ret)
+				return ret;
+		}
+
+	}
+	else {
+		dev_map = dev_map_lookup(rd, false);
+		if (!dev_map) {
+			pr_err("Wrong device name\n");
+			return -ENOENT;
+		}
+		rd_arg_inc(rd);
+		rd->dev_idx = dev_map->idx;
+		ret = dev_one_show(rd);
+	}
+	return ret;
+}
+
+int cmd_dev(struct rdma *rd)
+{
+	const struct rdma_cmd cmds[] = {
+		{ NULL,		dev_show },
+		{ "show",	dev_show },
+		{ "list",	dev_show },
+		{ "help",	dev_help },
+		{ 0 }
+	};
+
+	return rdma_exec_cmd(rd, cmds, "dev command");
+}
diff --git a/rdma/rdma.c b/rdma/rdma.c
index 29273839..dfebd71e 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c
@@ -18,7 +18,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { help }\n"
+	       "where  OBJECT := { dev | help }\n"
 	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
 }

@@ -33,6 +33,7 @@ static int rd_cmd(struct rdma *rd)
 	const struct rdma_cmd cmds[] = {
 		{ NULL,		cmd_help },
 		{ "help",	cmd_help },
+		{ "dev",	cmd_dev },
 		{ 0 }
 	};

diff --git a/rdma/rdma.h b/rdma/rdma.h
index 0c01b091..2856e592 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h
@@ -16,7 +16,6 @@
 #include <errno.h>
 #include <getopt.h>
 #include <libmnl/libmnl.h>
-
 #include <rdma/rdma_netlink.h>
 #include "list.h"

@@ -36,6 +35,8 @@ struct rdma {
 	char *filename;
 	bool show_details;
 	struct list_head dev_map_list;
+	uint32_t dev_idx;
+	uint32_t port_idx;
 	struct mnl_socket *nl;
 	struct nlmsghdr *nlh;
 	char *buff;
@@ -51,13 +52,22 @@ struct rdma_cmd {
  */
 bool rd_no_arg(struct rdma *rd);
 void rd_arg_inc(struct rdma *rd);
+char *rd_argv(struct rdma *rd);
+uint32_t get_port_from_argv(struct rdma *rd);

+/*
+ * Commands interface
+ */
+int cmd_dev(struct rdma *rd);
+int cmd_link(struct rdma *rd);
 int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *c, const char *str);

 /*
  * Device manipulation
  */
 void rdma_free_devmap(struct rdma *rd);
+struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index);
+struct dev_map *_dev_map_lookup(struct rdma *rd, const char *dev_name);

 /*
  * Netlink
diff --git a/rdma/utils.c b/rdma/utils.c
index b554eff7..bee490da 100644
--- a/rdma/utils.c
+++ b/rdma/utils.c
@@ -21,7 +21,7 @@ static int rd_argc(struct rdma *rd)
 	return rd->argc;
 }

-static char *rd_argv(struct rdma *rd)
+char *rd_argv(struct rdma *rd)
 {
 	if (!rd_argc(rd))
 		return NULL;
@@ -55,6 +55,15 @@ bool rd_no_arg(struct rdma *rd)
 	return rd_argc(rd) == 0;
 }

+uint32_t get_port_from_argv(struct rdma *rd)
+{
+        char *slash;
+
+        slash = strchr(rd_argv(rd), '/');
+        /* if no port found, return 0 */
+        return (slash) ? (atoi(slash + 1)):0;
+}
+
 static struct dev_map *dev_map_alloc(const char *dev_name)
 {
 	struct dev_map *dev_map;
@@ -88,8 +97,14 @@ static void dev_map_cleanup(struct rdma *rd)
 }

 static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
+	[RDMA_NLDEV_ATTR_DEV_INDEX] = MNL_TYPE_U32,
 	[RDMA_NLDEV_ATTR_DEV_NAME] = MNL_TYPE_NUL_STRING,
 	[RDMA_NLDEV_ATTR_PORT_INDEX] = MNL_TYPE_U32,
+	[RDMA_NLDEV_ATTR_CAP_FLAGS] = MNL_TYPE_U64,
+	[RDMA_NLDEV_ATTR_FW_VERSION] = MNL_TYPE_NUL_STRING,
+	[RDMA_NLDEV_ATTR_NODE_GUID] = MNL_TYPE_U64,
+	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = MNL_TYPE_U64,
+	[RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8,
 };

 int rd_attr_cb(const struct nlattr *attr, void *data)
@@ -134,7 +149,6 @@ int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)

 	dev_map->num_ports = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
 	dev_map->idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
-
 	return MNL_CB_OK;
 }

@@ -221,3 +235,31 @@ int rdma_recv_msg(struct rdma *rd, mnl_cb_t callback, void *data, unsigned int s
 	mnl_socket_close(rd->nl);
 	return ret;
 }
+
+struct dev_map *_dev_map_lookup(struct rdma *rd, const char *dev_name)
+{
+	struct dev_map *dev_map;
+
+	list_for_each_entry(dev_map, &rd->dev_map_list, list)
+		if (strcmp(dev_name, dev_map->dev_name) == 0)
+			return dev_map;
+
+	return NULL;
+}
+struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index)
+{
+	struct dev_map *dev_map;
+	char *dev_name;
+	char *slash;
+
+	dev_name = strdup(rd_argv(rd));
+	if (allow_port_index) {
+		slash = strrchr(dev_name, '/');
+		if (slash)
+			*slash = '\0';
+	}
+
+	dev_map = _dev_map_lookup(rd, dev_name);
+	free(dev_name);
+	return dev_map;
+}

[PATCH iproute2 V3 3/4] rdma: Add link object

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

Link (port) object represent struct ib_port to the user space.

Link properties:
 * Port capabilities
 * IB subnet prefix
 * LID, SM_LID and LMC
 * Port state
 * Physical state

Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 rdma/Makefile |   2 +-
 rdma/link.c   | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.c   |   3 +-
 rdma/utils.c  |   5 ++
 4 files changed, 288 insertions(+), 2 deletions(-)
 create mode 100644 rdma/link.c

diff --git a/rdma/Makefile b/rdma/Makefile
index 123d7ac5..1a9e4b1a 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile
@@ -2,7 +2,7 @@ include ../Config

 ifeq ($(HAVE_MNL),y)

-RDMA_OBJ = rdma.o utils.o dev.o
+RDMA_OBJ = rdma.o utils.o dev.o link.o

 TARGETS=rdma
 CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
diff --git a/rdma/link.c b/rdma/link.c
new file mode 100644
index 00000000..f92b4cef
--- /dev/null
+++ b/rdma/link.c
@@ -0,0 +1,280 @@
+/*
+ * link.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static int link_help(struct rdma *rd)
+{
+	pr_out("Usage: %s link show [DEV/PORT_INDEX]\n", rd->filename);
+	return 0;
+}
+
+static void link_print_caps(struct nlattr **tb)
+{
+	uint64_t caps;
+	uint32_t idx;
+
+	/*
+	 * FIXME: move to indexes when kernel will start exporting them.
+	 */
+	static const char *link_caps[64] = {
+		"UNKNOWN",
+		"SM",
+		"NOTICE",
+		"TRAP",
+		"OPT_IPD",
+		"AUTO_MIGR",
+		"SL_MAP",
+		"MKEY_NVRAM",
+		"PKEY_NVRAM",
+		"LED_INFO",
+		"SM_DISABLED",
+		"SYS_IMAGE_GUID",
+		"PKEY_SW_EXT_PORT_TRAP",
+		"UNKNOWN",
+		"EXTENDED_SPEEDS",
+		"UNKNOWN",
+		"CM",
+		"SNMP_TUNNEL",
+		"REINIT",
+		"DEVICE_MGMT",
+		"VENDOR_CLASS",
+		"DR_NOTICE",
+		"CAP_MASK_NOTICE",
+		"BOOT_MGMT",
+		"LINK_LATENCY",
+		"CLIENT_REG",
+		"IP_BASED_GIDS",
+	};
+
+	if (!tb[RDMA_NLDEV_ATTR_CAP_FLAGS])
+		return;
+
+	caps = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_CAP_FLAGS]);
+
+	pr_out("\n    caps: <");
+	for (idx = 0; idx < 64; idx++) {
+		if (caps & 0x1) {
+			pr_out("%s", link_caps[idx]?link_caps[idx]:"UNKNONW");
+			if (caps >> 0x1)
+				pr_out(", ");
+		}
+		caps >>= 0x1;
+	}
+
+	pr_out(">");
+}
+
+static void link_print_subnet_prefix(struct nlattr **tb)
+{
+	uint64_t subnet_prefix;
+	uint16_t sp[4];
+
+	if (!tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX])
+		return;
+
+	subnet_prefix = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX]);
+	memcpy(sp, &subnet_prefix, sizeof(uint64_t));
+	pr_out("subnet_prefix %04x:%04x:%04x:%04x ", sp[3], sp[2], sp[1], sp[0]);
+}
+
+static void link_print_lid(struct nlattr **tb)
+{
+	if (!tb[RDMA_NLDEV_ATTR_LID])
+		return;
+
+	pr_out("lid %u ",
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_LID]));
+}
+
+static void link_print_sm_lid(struct nlattr **tb)
+{
+
+	if (!tb[RDMA_NLDEV_ATTR_SM_LID])
+		return;
+
+	pr_out("sm_lid %u ",
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_SM_LID]));
+}
+
+static void link_print_lmc(struct nlattr **tb)
+{
+	if (!tb[RDMA_NLDEV_ATTR_LMC])
+		return;
+
+	pr_out("lmc %u ", mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_LMC]));
+}
+
+static void link_print_state(struct nlattr **tb)
+{
+	uint8_t state;
+	/*
+	 * FIXME: move to index exported by the kernel
+	 */
+	static const char *str[] = {
+		"NOP",
+		"DOWN",
+		"INIT",
+		"ARMED",
+		"ACTIVE",
+		"ACTIVE_DEFER",
+	};
+
+	if (!tb[RDMA_NLDEV_ATTR_PORT_STATE])
+		return;
+
+	state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_STATE]);
+
+	if (state < 6 )
+		pr_out("state %s ", str[state]);
+	else
+		pr_out("state UNKNOWN ");
+}
+
+static void link_print_phys_state(struct nlattr **tb)
+{
+	uint8_t phys_state;
+	/*
+	 * FIXME: move to index exported by the kernel
+	 */
+	static const char *str[] = {
+		"UNKNOWN",
+		"SLEEP",
+		"POLLING",
+		"DISABLED",
+		"PORT_CONFIGURATION_TRAINING",
+		"LINK_UP",
+		"LINK_ERROR_RECOVER",
+		"PHY_TEST",
+	};
+
+	if (!tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE])
+		return;
+
+	phys_state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE]);
+	if (phys_state < 8)
+		pr_out("physical_state %s ", str[phys_state]);
+	else
+		pr_out("physical_state UNKNOWN ");
+}
+
+static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
+{
+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
+	struct rdma *rd = data;
+
+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
+	if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_DEV_NAME])
+		return MNL_CB_ERROR;
+
+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
+		pr_err("This tool doesn't support switches yet\n");
+		return MNL_CB_ERROR;
+	}
+
+	pr_out("%u/%u: %s/%u: ",
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]),
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]),
+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]),
+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]));
+	link_print_subnet_prefix(tb);
+	link_print_lid(tb);
+	link_print_sm_lid(tb);
+	link_print_lmc(tb);
+	link_print_state(tb);
+	link_print_phys_state(tb);
+	if (rd->show_details)
+		link_print_caps(tb);
+
+	pr_out("\n");
+	return MNL_CB_OK;
+}
+
+static int link_no_args(struct rdma *rd)
+{
+	uint32_t seq;
+	int ret;
+
+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_PORT_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK));
+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_PORT_INDEX, rd->port_idx);
+	if ((ret = rdma_send_msg(rd)))
+		return ret;
+
+	return rdma_recv_msg(rd, link_parse_cb, rd, seq);
+}
+
+static int link_one_show(struct rdma *rd)
+{
+	const struct rdma_cmd cmds[] = {
+		{ NULL,		link_no_args},
+		{ 0 }
+	};
+
+	return rdma_exec_cmd(rd, cmds, "parameter");
+
+}
+
+static int link_show(struct rdma *rd)
+{
+	struct dev_map *dev_map;
+	uint32_t port;
+	int ret = 0;
+
+	if (rd_no_arg(rd)) {
+		list_for_each_entry(dev_map, &rd->dev_map_list, list) {
+			rd->dev_idx = dev_map->idx;
+			for (port = 1; port < dev_map->num_ports + 1; port++) {
+				rd->port_idx = port;
+				ret = link_one_show(rd);
+				if (ret)
+					return ret;
+			}
+		}
+
+	}
+	else {
+		dev_map = dev_map_lookup(rd, true);
+		port = get_port_from_argv(rd);
+		if (!dev_map || port > dev_map->num_ports) {
+			pr_err("Wrong device name\n");
+			return -ENOENT;
+		}
+		rd_arg_inc(rd);
+		rd->port_idx = port ? :1;
+		for (; port < dev_map->num_ports + 1; port++, rd->port_idx++) {
+			ret = link_one_show(rd);
+			if (ret)
+				return ret;
+			if (port)
+				/*
+				 * We got request to show link for devname
+				 * without port index.
+				 */
+				break;
+		}
+
+	}
+	return ret;
+}
+
+int cmd_link(struct rdma *rd)
+{
+	const struct rdma_cmd cmds[] = {
+		{ NULL,		link_show },
+		{ "show",	link_show },
+		{ "list",	link_show },
+		{ "help",	link_help },
+		{ 0 }
+	};
+
+	return rdma_exec_cmd(rd, cmds, "link command");
+}
diff --git a/rdma/rdma.c b/rdma/rdma.c
index dfebd71e..f597f614 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c
@@ -18,7 +18,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { dev | help }\n"
+	       "where  OBJECT := { dev | link | help }\n"
 	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
 }

@@ -34,6 +34,7 @@ static int rd_cmd(struct rdma *rd)
 		{ NULL,		cmd_help },
 		{ "help",	cmd_help },
 		{ "dev",	cmd_dev },
+		{ "link",	cmd_link },
 		{ 0 }
 	};

diff --git a/rdma/utils.c b/rdma/utils.c
index bee490da..ca61ccc1 100644
--- a/rdma/utils.c
+++ b/rdma/utils.c
@@ -104,6 +104,11 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 	[RDMA_NLDEV_ATTR_FW_VERSION] = MNL_TYPE_NUL_STRING,
 	[RDMA_NLDEV_ATTR_NODE_GUID] = MNL_TYPE_U64,
 	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = MNL_TYPE_U64,
+	[RDMA_NLDEV_ATTR_LID] = MNL_TYPE_U32,
+	[RDMA_NLDEV_ATTR_SM_LID] = MNL_TYPE_U32,
+	[RDMA_NLDEV_ATTR_LMC] = MNL_TYPE_U8,
+	[RDMA_NLDEV_ATTR_PORT_STATE] = MNL_TYPE_U8,
+	[RDMA_NLDEV_ATTR_PORT_PHYS_STATE] = MNL_TYPE_U8,
 	[RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8,
 };

[PATCH iproute2 V3 4/4] rdma: Add initial manual for the tool

[Leon Romanovsky]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 man/man8/Makefile |  3 +-
 man/man8/rdma.8   | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 man/man8/rdma.8

diff --git a/man/man8/Makefile b/man/man8/Makefile
index f3318644..81979a07 100644
--- a/man/man8/Makefile
+++ b/man/man8/Makefile
@@ -19,7 +19,8 @@ MAN8PAGES = $(TARGETS) ip.8 arpd.8 lnstat.8 routel.8 rtacct.8 rtmon.8 rtpr.8 ss.
 	tc-simple.8 tc-skbedit.8 tc-vlan.8 tc-xt.8 tc-skbmod.8 tc-ife.8 \
 	tc-tunnel_key.8 tc-sample.8 \
 	devlink.8 devlink-dev.8 devlink-monitor.8 devlink-port.8 devlink-sb.8 \
-	ifstat.8
+	ifstat.8 \
+	rdma.8

 all: $(TARGETS)

diff --git a/man/man8/rdma.8 b/man/man8/rdma.8
new file mode 100644
index 00000000..7578c15e
--- /dev/null
+++ b/man/man8/rdma.8
@@ -0,0 +1,82 @@
+.TH RDMA 8 "28 Mar 2017" "iproute2" "Linux"
+.SH NAME
+rdma \- RDMA tool
+.SH SYNOPSIS
+.sp
+.ad l
+.in +8
+.ti -8
+.B rdma
+.RI "[ " OPTIONS " ] " OBJECT " { " COMMAND " | "
+.BR help " }"
+.sp
+
+.ti -8
+.IR OBJECT " := { "
+.BR dev " | " link " }"
+.sp
+
+.ti -8
+.IR OPTIONS " := { "
+\fB\-V\fR[\fIersion\fR] }
+
+.SH OPTIONS
+
+.TP
+.BR "\-V" , " -Version"
+Print the version of the
+.B rdma
+tool and exit.
+
+.SS
+.I OBJECT
+
+.TP
+.B dev
+- RDMA device.
+
+.TP
+.B link
+- RDMA port related.
+
+.PP
+The names of all objects may be written in full or
+abbreviated form, for example
+.B stats
+can be abbreviated as
+.B stat
+or just
+.B s.
+
+.SS
+.I COMMAND
+
+Specifies the action to perform on the object.
+The set of possible actions depends on the object type.
+As a rule, it is possible to
+.B show
+(or
+.B list
+) objects, but some objects do not allow all of these operations
+or have some additional commands. The
+.B help
+command is available for all objects. It prints
+out a list of available commands and argument syntax conventions.
+.sp
+If no command is given, some default command is assumed.
+Usually it is
+.B list
+or, if the objects of this class cannot be listed,
+.BR "help" .
+
+.SH EXIT STATUS
+Exit status is 0 if command was successful or a positive integer upon failure.
+
+.SH REPORTING BUGS
+Report any bugs to the Linux RDMA mailing list
+.B <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
+where the development and maintenance is primarily done.
+You do not have to be subscribed to the list to send a message there.
+
+.SH AUTHOR
+Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
--
2.13.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH iproute2 V3 2/4] rdma: Add dev object

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On Tue, Jul 04, 2017 at 10:55:39AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
>
> Device (dev) object represents struct ib_device to the user space.
>
> Device properties:
>  * Device capabilities
>  * FW version to the device output
>  * node_guid and sys_image_guid
>  * node_type
>
> Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> ---

<...>

> +}
> +
> +static void dev_print_node_type(struct nlattr **tb)
> +{
> +	uint8_t node_type;
> +	/*
> +	 * FIXME: move to index exported by the kernel
> +	 */
> +	static const char *str[] = {
> +		"UNKNOWN",
> +		"SWITCH",
> +		"ROUTER",
> +		"RNIC",
> +		"USNIC",
> +		"USNIC_UDP",
> +	};

There is a bug here, i forgot to add "CA" to the list of node_types
(before SWITCH). Thanks to Or Gerlitz who spotted it.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 0/4] RDMAtool

[Jiri Pirko]

Tue, Jul 04, 2017 at 09:55:37AM CEST, leon@kernel.org wrote:
>Hi,
>
>This is third version of series implementing the RDAMtool -  the tool

s/RDAMtool/RDMAtool/


>to configure RDMA devices. The initial proposal was sent as RFC [1] and
>was based on sysfs entries as POC.
>
>The current series was rewritten completely to work with RDMA netlinks as
>a source of user<->kernel communications. In order to achieve that, the
>RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].
>
>The following is an example of various runs on my machine with 5 devices
>(4 in IB mode and one in Ethernet mode)
>
>### Without parameters
>$ rdma
>Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
>where  OBJECT := { dev | link | help }
>       OPTIONS := { -V[ersion] | -d[etails]}

Do you plan to add "mon" any time soon?

Re: [PATCH iproute2 V3 1/4] rdma: Add basic infrastructure for RDMA tool

[Jiri Pirko]

Tue, Jul 04, 2017 at 09:55:38AM CEST, leon@kernel.org wrote:
>From: Leon Romanovsky <leonro@mellanox.com>
>
>RDMA devices are cross-functional devices from one side,
>but very tailored for the specific markets from another.
>
>Such diversity caused to spread of RDMA related configuration
>across various tools, e.g. devlink, ip, ethtool, ib specific and
>vendor specific solutions.
>
>This patch adds ability to fill device and port information
>by reading RDMA netlink.
>
>Signed-off-by: Leon Romanovsky <leonro@mellanox.com>

[...]

	
>+
>+int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)
>+{
>+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
>+	struct dev_map *dev_map;
>+	struct rdma *rd = data;
>+	const char *dev_name;
>+
>+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
>+	if (!tb[RDMA_NLDEV_ATTR_DEV_NAME] || !tb[RDMA_NLDEV_ATTR_DEV_INDEX])
>+		return MNL_CB_ERROR;
>+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
>+		pr_err("This tool doesn't support switches yet\n");
>+		return MNL_CB_ERROR;
>+	}
>+
>+	dev_name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
>+
>+	dev_map = dev_map_alloc(dev_name);
>+	if (!dev_map)
>+		/* The main function will cleanup the allocations */
>+		return MNL_CB_ERROR;
>+	list_add_tail(&dev_map->list, &rd->dev_map_list);
>+
>+	dev_map->num_ports = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
>+	dev_map->idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
>+
>+	return MNL_CB_OK;
>+}
>+
>+void rdma_free_devmap(struct rdma *rd)
>+{
>+	if(!rd)
>+		return;
>+	dev_map_cleanup(rd);
>+}
>+
>+int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *cmds, const char *str)

Sometimes the functions are prefixed "rd_", sometimes "rdma_".
I think it would be nice to unify this with "rd_".


>+{
>+	const struct rdma_cmd *c;
>+
>+	/* First argument in objs table is default variant */
>+	if (rd_no_arg(rd))
>+		return cmds->func(rd);
>+
>+	for (c = cmds + 1; c->cmd; ++c) {
>+		if (rd_argv_match(rd, c->cmd)) {
>+			/* Move to next argument */
>+			rd_arg_inc(rd);
>+			return c->func(rd);
>+		}
>+	}
>+
>+	pr_err("Unknown %s '%s'.\n", str, rd_argv(rd));
>+	return 0;
>+}
>+

Re: [PATCH iproute2 V3 2/4] rdma: Add dev object

[Jiri Pirko]

Tue, Jul 04, 2017 at 09:55:39AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
>From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>
>Device (dev) object represents struct ib_device to the user space.
>
>Device properties:
> * Device capabilities
> * FW version to the device output
> * node_guid and sys_image_guid
> * node_type
>
>Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>---
> rdma/Makefile |   2 +-
> rdma/dev.c    | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> rdma/rdma.c   |   3 +-
> rdma/rdma.h   |  12 ++-
> rdma/utils.c  |  46 +++++++++++-
> 5 files changed, 293 insertions(+), 5 deletions(-)
> create mode 100644 rdma/dev.c
>
>diff --git a/rdma/Makefile b/rdma/Makefile
>index 64da2142..123d7ac5 100644
>--- a/rdma/Makefile
>+++ b/rdma/Makefile
>@@ -2,7 +2,7 @@ include ../Config
>
> ifeq ($(HAVE_MNL),y)
>
>-RDMA_OBJ = rdma.o utils.o
>+RDMA_OBJ = rdma.o utils.o dev.o
>
> TARGETS=rdma
> CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
>diff --git a/rdma/dev.c b/rdma/dev.c
>new file mode 100644
>index 00000000..b80e5288
>--- /dev/null
>+++ b/rdma/dev.c
>@@ -0,0 +1,235 @@
>+/*
>+ * dev.c	RDMA tool
>+ *
>+ *              This program is free software; you can redistribute it and/or
>+ *              modify it under the terms of the GNU General Public License
>+ *              as published by the Free Software Foundation; either version
>+ *              2 of the License, or (at your option) any later version.
>+ *
>+ * Authors:     Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>+ */
>+
>+#include "rdma.h"
>+
>+static int dev_help(struct rdma *rd)
>+{
>+	pr_out("Usage: %s dev show [DEV]\n", rd->filename);
>+	return 0;
>+}
>+
>+static void dev_print_caps(struct nlattr **tb)
>+{
>+	uint64_t caps;
>+	uint32_t idx;
>+
>+	/*
>+	 * FIXME: move to indexes when kernel will start exporting them.
>+	 */
>+	static const char *dev_caps[64] = {

	static const char *dev_caps[] = {


>+		"RESIZE_MAX_WR",
>+		"BAD_PKEY_CNTR",
>+		"BAD_QKEY_CNTR",
>+		"RAW_MULTI",
>+		"AUTO_PATH_MIG",
>+		"CHANGE_PHY_PORT",
>+		"UD_AV_PORT_ENFORCE",
>+		"CURR_QP_STATE_MOD",
>+		"SHUTDOWN_PORT",
>+		"INIT_TYPE",
>+		"PORT_ACTIVE_EVENT",
>+		"SYS_IMAGE_GUID",
>+		"RC_RNR_NAK_GEN",
>+		"SRQ_RESIZE",
>+		"N_NOTIFY_CQ",
>+		"LOCAL_DMA_LKEY",
>+		"RESERVED",
>+		"MEM_WINDOW",
>+		"UD_IP_CSUM",
>+		"UD_TSO",
>+		"XRC",
>+		"MEM_MGT_EXTENSIONS",
>+		"BLOCK_MULTICAST_LOOPBACK",
>+		"MEM_WINDOW_TYPE_2A",
>+		"MEM_WINDOW_TYPE_2B",
>+		"RC_IP_CSUM",
>+		"RAW_IP_CSUM",
>+		"CROSS_CHANNEL",
>+		"MANAGED_FLOW_STEERING",
>+		"SIGNATURE_HANDOVER",
>+		"ON_DEMAND_PAGING",
>+		"SG_GAPS_REG",
>+		"VIRTUAL_FUNCTION",
>+		"RAW_SCATTER_FCS",
>+		"RDMA_NETDEV_OPA_VNIC",
>+	};
>+
>+	if (!tb[RDMA_NLDEV_ATTR_CAP_FLAGS])
>+	       return;
>+
>+	caps = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_CAP_FLAGS]);
>+
>+	pr_out("\n    caps: <");
>+	for (idx = 0; idx < 64; idx++) {
>+		if (caps & 0x1) {
>+			pr_out("%s", dev_caps[idx]?dev_caps[idx]:"UNKNONW");
>+			if (caps >> 0x1)
>+				pr_out(", ");
>+		}
>+		caps >>= 0x1;
>+	}
>+
>+	pr_out(">");
>+}
>+
>+static void dev_print_fw(struct nlattr **tb)
>+{
>+	if (!tb[RDMA_NLDEV_ATTR_FW_VERSION])
>+		return;
>+
>+	pr_out("fw %s ",
>+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_FW_VERSION]));
>+}
>+
>+static void _dev_print_be64(char *name, uint64_t val)

Why the initial "_"? I think it could be avoided.

Wait, I'm confused. Does kernel actually exposes BE64 in U64 attribute?
That is wrong. Please fix it while there is still a time (if it is).


>+{
>+	uint16_t vp[4];
>+
>+	memcpy(vp, &val, sizeof(uint64_t));
>+	pr_out("%s %04x:%04x:%04x:%04x ", name, vp[3], vp[2], vp[1], vp[0]);
>+}
>+
>+static void dev_print_node_guid(struct nlattr **tb)
>+{
>+	uint64_t node_guid;
>+
>+	if (!tb[RDMA_NLDEV_ATTR_NODE_GUID])
>+		return;
>+
>+	node_guid = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_NODE_GUID]);
>+	_dev_print_be64("node_guid", node_guid);
>+}
>+
>+static void dev_print_sys_image_guid(struct nlattr **tb)
>+{
>+       uint64_t	sys_image_guid;
>+
>+	if (!tb[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID])
>+		return;
>+
>+	sys_image_guid = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID]);
>+	_dev_print_be64("sys_image_guid", sys_image_guid);
>+}
>+
>+static void dev_print_node_type(struct nlattr **tb)
>+{
>+	uint8_t node_type;
>+	/*
>+	 * FIXME: move to index exported by the kernel
>+	 */

So fix it rightaway :)


>+	static const char *str[] = {
>+		"UNKNOWN",
>+		"SWITCH",
>+		"ROUTER",
>+		"RNIC",
>+		"USNIC",
>+		"USNIC_UDP",
>+	};
>+
>+	if (!tb[RDMA_NLDEV_ATTR_DEV_NODE_TYPE])
>+		return;
>+
>+	node_type = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_DEV_NODE_TYPE]);
>+
>+	if (node_type < 7 )

Magic? :O


>+		pr_out("node_type %s ", str[node_type]);
>+	else
>+		pr_out("node_type UNKNOWN ");
>+}
>+
>+static int dev_parse_cb(const struct nlmsghdr *nlh, void *data)
>+{
>+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
>+	struct rdma *rd = data;
>+
>+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
>+	if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_DEV_NAME])
>+		return MNL_CB_ERROR;
>+
>+	pr_out("%u: %s: ",
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]),
>+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]));
>+	dev_print_node_type(tb);
>+	dev_print_fw(tb);
>+	dev_print_node_guid(tb);
>+	dev_print_sys_image_guid(tb);
>+	if (rd->show_details)
>+		dev_print_caps(tb);
>+
>+	pr_out("\n");
>+	return MNL_CB_OK;
>+}
>+
>+
>+static int dev_no_args(struct rdma *rd)
>+{
>+	uint32_t seq;
>+	int ret;
>+
>+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK));
>+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
>+	if ((ret = rdma_send_msg(rd)))
>+		return ret;
>+
>+	return rdma_recv_msg(rd, dev_parse_cb, rd, seq);
>+}
>+
>+static int dev_one_show(struct rdma *rd)
>+{
>+	const struct rdma_cmd cmds[] = {
>+		{ NULL,		dev_no_args},
>+		{ 0 }
>+	};
>+
>+	return rdma_exec_cmd(rd, cmds, "parameter");
>+

Drop the extra empty line.


>+}
>+
>+static int dev_show(struct rdma *rd)
>+{
>+	struct dev_map *dev_map;
>+	int ret = 0;
>+
>+	if (rd_no_arg(rd)) {
>+		list_for_each_entry(dev_map, &rd->dev_map_list, list) {
>+			rd->dev_idx = dev_map->idx;
>+			ret = dev_one_show(rd);
>+			if (ret)
>+				return ret;
>+		}
>+

Drop the extra empty line.


>+	}
>+	else {

"} else {"

Checkpatch should warn you. Did you run it?



>+		dev_map = dev_map_lookup(rd, false);
>+		if (!dev_map) {
>+			pr_err("Wrong device name\n");
>+			return -ENOENT;
>+		}
>+		rd_arg_inc(rd);
>+		rd->dev_idx = dev_map->idx;
>+		ret = dev_one_show(rd);
>+	}
>+	return ret;
>+}
>+
>+int cmd_dev(struct rdma *rd)
>+{
>+	const struct rdma_cmd cmds[] = {
>+		{ NULL,		dev_show },
>+		{ "show",	dev_show },
>+		{ "list",	dev_show },
>+		{ "help",	dev_help },
>+		{ 0 }
>+	};
>+
>+	return rdma_exec_cmd(rd, cmds, "dev command");
>+}
>diff --git a/rdma/rdma.c b/rdma/rdma.c
>index 29273839..dfebd71e 100644
>--- a/rdma/rdma.c
>+++ b/rdma/rdma.c
>@@ -18,7 +18,7 @@
> static void help(char *name)
> {
> 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
>-	       "where  OBJECT := { help }\n"
>+	       "where  OBJECT := { dev | help }\n"
> 	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
> }
>
>@@ -33,6 +33,7 @@ static int rd_cmd(struct rdma *rd)
> 	const struct rdma_cmd cmds[] = {
> 		{ NULL,		cmd_help },
> 		{ "help",	cmd_help },
>+		{ "dev",	cmd_dev },
> 		{ 0 }
> 	};
>
>diff --git a/rdma/rdma.h b/rdma/rdma.h
>index 0c01b091..2856e592 100644
>--- a/rdma/rdma.h
>+++ b/rdma/rdma.h
>@@ -16,7 +16,6 @@
> #include <errno.h>
> #include <getopt.h>
> #include <libmnl/libmnl.h>
>-

:) Please push this line removal to the previous patch.


> #include <rdma/rdma_netlink.h>
> #include "list.h"
>
>@@ -36,6 +35,8 @@ struct rdma {
> 	char *filename;
> 	bool show_details;
> 	struct list_head dev_map_list;
>+	uint32_t dev_idx;
>+	uint32_t port_idx;
> 	struct mnl_socket *nl;
> 	struct nlmsghdr *nlh;
> 	char *buff;
>@@ -51,13 +52,22 @@ struct rdma_cmd {
>  */
> bool rd_no_arg(struct rdma *rd);
> void rd_arg_inc(struct rdma *rd);
>+char *rd_argv(struct rdma *rd);
>+uint32_t get_port_from_argv(struct rdma *rd);
>
>+/*
>+ * Commands interface
>+ */
>+int cmd_dev(struct rdma *rd);
>+int cmd_link(struct rdma *rd);
> int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *c, const char *str);
>
> /*
>  * Device manipulation
>  */
> void rdma_free_devmap(struct rdma *rd);
>+struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index);
>+struct dev_map *_dev_map_lookup(struct rdma *rd, const char *dev_name);
>
> /*
>  * Netlink
>diff --git a/rdma/utils.c b/rdma/utils.c
>index b554eff7..bee490da 100644
>--- a/rdma/utils.c
>+++ b/rdma/utils.c
>@@ -21,7 +21,7 @@ static int rd_argc(struct rdma *rd)
> 	return rd->argc;
> }
>
>-static char *rd_argv(struct rdma *rd)
>+char *rd_argv(struct rdma *rd)
> {
> 	if (!rd_argc(rd))
> 		return NULL;
>@@ -55,6 +55,15 @@ bool rd_no_arg(struct rdma *rd)
> 	return rd_argc(rd) == 0;
> }
>
>+uint32_t get_port_from_argv(struct rdma *rd)
>+{
>+        char *slash;
>+
>+        slash = strchr(rd_argv(rd), '/');
>+        /* if no port found, return 0 */
>+        return (slash) ? (atoi(slash + 1)):0;

	return slash ? atoi(slash + 1) : 0;



>+}
>+
> static struct dev_map *dev_map_alloc(const char *dev_name)
> {
> 	struct dev_map *dev_map;
>@@ -88,8 +97,14 @@ static void dev_map_cleanup(struct rdma *rd)
> }
>
> static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
>+	[RDMA_NLDEV_ATTR_DEV_INDEX] = MNL_TYPE_U32,
> 	[RDMA_NLDEV_ATTR_DEV_NAME] = MNL_TYPE_NUL_STRING,
> 	[RDMA_NLDEV_ATTR_PORT_INDEX] = MNL_TYPE_U32,
>+	[RDMA_NLDEV_ATTR_CAP_FLAGS] = MNL_TYPE_U64,
>+	[RDMA_NLDEV_ATTR_FW_VERSION] = MNL_TYPE_NUL_STRING,
>+	[RDMA_NLDEV_ATTR_NODE_GUID] = MNL_TYPE_U64,
>+	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = MNL_TYPE_U64,
>+	[RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8,
> };
>
> int rd_attr_cb(const struct nlattr *attr, void *data)
>@@ -134,7 +149,6 @@ int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)
>
> 	dev_map->num_ports = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
> 	dev_map->idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
>-

Again, should be part of the previous patch.


> 	return MNL_CB_OK;
> }
>
>@@ -221,3 +235,31 @@ int rdma_recv_msg(struct rdma *rd, mnl_cb_t callback, void *data, unsigned int s
> 	mnl_socket_close(rd->nl);
> 	return ret;
> }
>+
>+struct dev_map *_dev_map_lookup(struct rdma *rd, const char *dev_name)
>+{
>+	struct dev_map *dev_map;
>+
>+	list_for_each_entry(dev_map, &rd->dev_map_list, list)
>+		if (strcmp(dev_name, dev_map->dev_name) == 0)
>+			return dev_map;
>+
>+	return NULL;
>+}
>+struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index)
>+{
>+	struct dev_map *dev_map;
>+	char *dev_name;
>+	char *slash;
>+
>+	dev_name = strdup(rd_argv(rd));
>+	if (allow_port_index) {
>+		slash = strrchr(dev_name, '/');
>+		if (slash)
>+			*slash = '\0';
>+	}
>+
>+	dev_map = _dev_map_lookup(rd, dev_name);
>+	free(dev_name);
>+	return dev_map;
>+}
>--
>2.13.2
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH iproute2 V3 0/4] RDMAtool

[Jiri Pirko]

Tue, Jul 04, 2017 at 09:55:37AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
>Hi,
>
>This is third version of series implementing the RDAMtool -  the tool
>to configure RDMA devices. The initial proposal was sent as RFC [1] and
>was based on sysfs entries as POC.
>
>The current series was rewritten completely to work with RDMA netlinks as
>a source of user<->kernel communications. In order to achieve that, the
>RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].
>
>The following is an example of various runs on my machine with 5 devices
>(4 in IB mode and one in Ethernet mode)
>
>### Without parameters
>$ rdma
>Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
>where  OBJECT := { dev | link | help }
>       OPTIONS := { -V[ersion] | -d[etails]}

What about json output? You will need it sooner than later. It will
prevent you from a lot of headaches if you implement it right away.
Lesson learned...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH iproute2 V3 3/4] rdma: Add link object

[Jiri Pirko]

Tue, Jul 04, 2017 at 09:55:40AM CEST, leon@kernel.org wrote:
>From: Leon Romanovsky <leonro@mellanox.com>
>
>Link (port) object represent struct ib_port to the user space.
>
>Link properties:
> * Port capabilities
> * IB subnet prefix
> * LID, SM_LID and LMC
> * Port state
> * Physical state
>
>Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
>---
> rdma/Makefile |   2 +-
> rdma/link.c   | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> rdma/rdma.c   |   3 +-
> rdma/utils.c  |   5 ++
> 4 files changed, 288 insertions(+), 2 deletions(-)
> create mode 100644 rdma/link.c
>
>diff --git a/rdma/Makefile b/rdma/Makefile
>index 123d7ac5..1a9e4b1a 100644
>--- a/rdma/Makefile
>+++ b/rdma/Makefile
>@@ -2,7 +2,7 @@ include ../Config
>
> ifeq ($(HAVE_MNL),y)
>
>-RDMA_OBJ = rdma.o utils.o dev.o
>+RDMA_OBJ = rdma.o utils.o dev.o link.o
>
> TARGETS=rdma
> CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
>diff --git a/rdma/link.c b/rdma/link.c
>new file mode 100644
>index 00000000..f92b4cef
>--- /dev/null
>+++ b/rdma/link.c
>@@ -0,0 +1,280 @@
>+/*
>+ * link.c	RDMA tool
>+ *
>+ *              This program is free software; you can redistribute it and/or
>+ *              modify it under the terms of the GNU General Public License
>+ *              as published by the Free Software Foundation; either version
>+ *              2 of the License, or (at your option) any later version.
>+ *
>+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
>+ */
>+
>+#include "rdma.h"
>+
>+static int link_help(struct rdma *rd)
>+{
>+	pr_out("Usage: %s link show [DEV/PORT_INDEX]\n", rd->filename);
>+	return 0;
>+}
>+
>+static void link_print_caps(struct nlattr **tb)
>+{
>+	uint64_t caps;
>+	uint32_t idx;
>+
>+	/*
>+	 * FIXME: move to indexes when kernel will start exporting them.

Not exported yet?



>+	 */
>+	static const char *link_caps[64] = {

[]


>+		"UNKNOWN",
>+		"SM",
>+		"NOTICE",
>+		"TRAP",
>+		"OPT_IPD",
>+		"AUTO_MIGR",
>+		"SL_MAP",
>+		"MKEY_NVRAM",
>+		"PKEY_NVRAM",
>+		"LED_INFO",
>+		"SM_DISABLED",
>+		"SYS_IMAGE_GUID",
>+		"PKEY_SW_EXT_PORT_TRAP",
>+		"UNKNOWN",
>+		"EXTENDED_SPEEDS",
>+		"UNKNOWN",
>+		"CM",
>+		"SNMP_TUNNEL",
>+		"REINIT",
>+		"DEVICE_MGMT",
>+		"VENDOR_CLASS",
>+		"DR_NOTICE",
>+		"CAP_MASK_NOTICE",
>+		"BOOT_MGMT",
>+		"LINK_LATENCY",
>+		"CLIENT_REG",
>+		"IP_BASED_GIDS",
>+	};
>+
>+	if (!tb[RDMA_NLDEV_ATTR_CAP_FLAGS])
>+		return;
>+
>+	caps = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_CAP_FLAGS]);
>+
>+	pr_out("\n    caps: <");
>+	for (idx = 0; idx < 64; idx++) {
>+		if (caps & 0x1) {
>+			pr_out("%s", link_caps[idx]?link_caps[idx]:"UNKNONW");

"link_caps[idx] ? link_caps[idx] : "UNKNOWN""

note the s/UNKNONW/UNKNOWN/


>+			if (caps >> 0x1)
>+				pr_out(", ");
>+		}
>+		caps >>= 0x1;

Interesting. 


>+	}
>+
>+	pr_out(">");
>+}
>+
>+static void link_print_subnet_prefix(struct nlattr **tb)
>+{
>+	uint64_t subnet_prefix;
>+	uint16_t sp[4];
>+
>+	if (!tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX])
>+		return;
>+
>+	subnet_prefix = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX]);
>+	memcpy(sp, &subnet_prefix, sizeof(uint64_t));
>+	pr_out("subnet_prefix %04x:%04x:%04x:%04x ", sp[3], sp[2], sp[1], sp[0]);

You have similar pr_out helper in the previous patch. Perhaps you can
re-use it?


>+}
>+
>+static void link_print_lid(struct nlattr **tb)
>+{
>+	if (!tb[RDMA_NLDEV_ATTR_LID])
>+		return;
>+
>+	pr_out("lid %u ",
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_LID]));
>+}
>+
>+static void link_print_sm_lid(struct nlattr **tb)
>+{
>+

Avoid the extra empty line.


>+	if (!tb[RDMA_NLDEV_ATTR_SM_LID])
>+		return;
>+
>+	pr_out("sm_lid %u ",
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_SM_LID]));
>+}
>+
>+static void link_print_lmc(struct nlattr **tb)
>+{
>+	if (!tb[RDMA_NLDEV_ATTR_LMC])
>+		return;
>+
>+	pr_out("lmc %u ", mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_LMC]));
>+}
>+
>+static void link_print_state(struct nlattr **tb)
>+{
>+	uint8_t state;
>+	/*
>+	 * FIXME: move to index exported by the kernel

Again, can't this be fixed now?


>+	 */
>+	static const char *str[] = {
>+		"NOP",
>+		"DOWN",
>+		"INIT",
>+		"ARMED",
>+		"ACTIVE",
>+		"ACTIVE_DEFER",
>+	};
>+
>+	if (!tb[RDMA_NLDEV_ATTR_PORT_STATE])
>+		return;
>+
>+	state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_STATE]);
>+
>+	if (state < 6 )

Magic?


>+		pr_out("state %s ", str[state]);
>+	else
>+		pr_out("state UNKNOWN ");
>+}
>+
>+static void link_print_phys_state(struct nlattr **tb)
>+{
>+	uint8_t phys_state;
>+	/*
>+	 * FIXME: move to index exported by the kernel

Again, can't this be fixed now?


>+	 */
>+	static const char *str[] = {
>+		"UNKNOWN",
>+		"SLEEP",
>+		"POLLING",
>+		"DISABLED",
>+		"PORT_CONFIGURATION_TRAINING",
>+		"LINK_UP",
>+		"LINK_ERROR_RECOVER",
>+		"PHY_TEST",
>+	};
>+
>+	if (!tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE])
>+		return;
>+
>+	phys_state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE]);
>+	if (phys_state < 8)

Magic?


>+		pr_out("physical_state %s ", str[phys_state]);
>+	else
>+		pr_out("physical_state UNKNOWN ");
>+}
>+
>+static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>+{
>+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
>+	struct rdma *rd = data;
>+
>+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
>+	if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_DEV_NAME])
>+		return MNL_CB_ERROR;
>+
>+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
>+		pr_err("This tool doesn't support switches yet\n");
>+		return MNL_CB_ERROR;
>+	}
>+
>+	pr_out("%u/%u: %s/%u: ",
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]),
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]),
>+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]),
>+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]));
>+	link_print_subnet_prefix(tb);
>+	link_print_lid(tb);
>+	link_print_sm_lid(tb);
>+	link_print_lmc(tb);
>+	link_print_state(tb);
>+	link_print_phys_state(tb);
>+	if (rd->show_details)
>+		link_print_caps(tb);
>+
>+	pr_out("\n");
>+	return MNL_CB_OK;
>+}
>+
>+static int link_no_args(struct rdma *rd)
>+{
>+	uint32_t seq;
>+	int ret;
>+
>+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_PORT_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK));
>+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
>+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_PORT_INDEX, rd->port_idx);
>+	if ((ret = rdma_send_msg(rd)))
>+		return ret;
>+
>+	return rdma_recv_msg(rd, link_parse_cb, rd, seq);
>+}
>+
>+static int link_one_show(struct rdma *rd)
>+{
>+	const struct rdma_cmd cmds[] = {
>+		{ NULL,		link_no_args},
>+		{ 0 }
>+	};
>+
>+	return rdma_exec_cmd(rd, cmds, "parameter");
>+

Avoid the extra empty line.


>+}
>+
>+static int link_show(struct rdma *rd)
>+{
>+	struct dev_map *dev_map;
>+	uint32_t port;
>+	int ret = 0;
>+
>+	if (rd_no_arg(rd)) {
>+		list_for_each_entry(dev_map, &rd->dev_map_list, list) {
>+			rd->dev_idx = dev_map->idx;
>+			for (port = 1; port < dev_map->num_ports + 1; port++) {
>+				rd->port_idx = port;
>+				ret = link_one_show(rd);
>+				if (ret)
>+					return ret;
>+			}
>+		}
>+
>+	}
>+	else {
>+		dev_map = dev_map_lookup(rd, true);
>+		port = get_port_from_argv(rd);
>+		if (!dev_map || port > dev_map->num_ports) {
>+			pr_err("Wrong device name\n");
>+			return -ENOENT;
>+		}
>+		rd_arg_inc(rd);
>+		rd->port_idx = port ? :1;

"port ? : 1"

>+		for (; port < dev_map->num_ports + 1; port++, rd->port_idx++) {
>+			ret = link_one_show(rd);
>+			if (ret)
>+				return ret;
>+			if (port)
>+				/*
>+				 * We got request to show link for devname
>+				 * without port index.
>+				 */
>+				break;
>+		}
>+
>+	}
>+	return ret;

	You can do return 0 here and avoid ret initialization.


>+}
>+
>+int cmd_link(struct rdma *rd)
>+{
>+	const struct rdma_cmd cmds[] = {
>+		{ NULL,		link_show },
>+		{ "show",	link_show },
>+		{ "list",	link_show },
>+		{ "help",	link_help },
>+		{ 0 }
>+	};
>+
>+	return rdma_exec_cmd(rd, cmds, "link command");
>+}
>diff --git a/rdma/rdma.c b/rdma/rdma.c
>index dfebd71e..f597f614 100644
>--- a/rdma/rdma.c
>+++ b/rdma/rdma.c
>@@ -18,7 +18,7 @@
> static void help(char *name)
> {
> 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
>-	       "where  OBJECT := { dev | help }\n"
>+	       "where  OBJECT := { dev | link | help }\n"
> 	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
> }
>
>@@ -34,6 +34,7 @@ static int rd_cmd(struct rdma *rd)
> 		{ NULL,		cmd_help },
> 		{ "help",	cmd_help },
> 		{ "dev",	cmd_dev },
>+		{ "link",	cmd_link },
> 		{ 0 }
> 	};
>
>diff --git a/rdma/utils.c b/rdma/utils.c
>index bee490da..ca61ccc1 100644
>--- a/rdma/utils.c
>+++ b/rdma/utils.c
>@@ -104,6 +104,11 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
> 	[RDMA_NLDEV_ATTR_FW_VERSION] = MNL_TYPE_NUL_STRING,
> 	[RDMA_NLDEV_ATTR_NODE_GUID] = MNL_TYPE_U64,
> 	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = MNL_TYPE_U64,
>+	[RDMA_NLDEV_ATTR_LID] = MNL_TYPE_U32,
>+	[RDMA_NLDEV_ATTR_SM_LID] = MNL_TYPE_U32,
>+	[RDMA_NLDEV_ATTR_LMC] = MNL_TYPE_U8,
>+	[RDMA_NLDEV_ATTR_PORT_STATE] = MNL_TYPE_U8,
>+	[RDMA_NLDEV_ATTR_PORT_PHYS_STATE] = MNL_TYPE_U8,
> 	[RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8,
> };
>
>--
>2.13.2
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH iproute2 V3 0/4] RDMAtool

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

On Mon, Jul 10, 2017 at 10:02:30AM +0200, Jiri Pirko wrote:
> Tue, Jul 04, 2017 at 09:55:37AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
> >Hi,
> >
> >This is third version of series implementing the RDAMtool -  the tool
> >to configure RDMA devices. The initial proposal was sent as RFC [1] and
> >was based on sysfs entries as POC.
> >
> >The current series was rewritten completely to work with RDMA netlinks as
> >a source of user<->kernel communications. In order to achieve that, the
> >RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].
> >
> >The following is an example of various runs on my machine with 5 devices
> >(4 in IB mode and one in Ethernet mode)
> >
> >### Without parameters
> >$ rdma
> >Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
> >where  OBJECT := { dev | link | help }
> >       OPTIONS := { -V[ersion] | -d[etails]}
>
> What about json output? You will need it sooner than later. It will
> prevent you from a lot of headaches if you implement it right away.
> Lesson learned...

I'm planning to do it in the coming kernel cycle.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 3/4] rdma: Add link object

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 10579 bytes --]

On Mon, Jul 10, 2017 at 10:13:07AM +0200, Jiri Pirko wrote:
> Tue, Jul 04, 2017 at 09:55:40AM CEST, leon@kernel.org wrote:
> >From: Leon Romanovsky <leonro@mellanox.com>
> >
> >Link (port) object represent struct ib_port to the user space.
> >
> >Link properties:
> > * Port capabilities
> > * IB subnet prefix
> > * LID, SM_LID and LMC
> > * Port state
> > * Physical state
> >
> >Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> >---
> > rdma/Makefile |   2 +-
> > rdma/link.c   | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > rdma/rdma.c   |   3 +-
> > rdma/utils.c  |   5 ++
> > 4 files changed, 288 insertions(+), 2 deletions(-)
> > create mode 100644 rdma/link.c
> >
> >diff --git a/rdma/Makefile b/rdma/Makefile
> >index 123d7ac5..1a9e4b1a 100644
> >--- a/rdma/Makefile
> >+++ b/rdma/Makefile
> >@@ -2,7 +2,7 @@ include ../Config
> >
> > ifeq ($(HAVE_MNL),y)
> >
> >-RDMA_OBJ = rdma.o utils.o dev.o
> >+RDMA_OBJ = rdma.o utils.o dev.o link.o
> >
> > TARGETS=rdma
> > CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
> >diff --git a/rdma/link.c b/rdma/link.c
> >new file mode 100644
> >index 00000000..f92b4cef
> >--- /dev/null
> >+++ b/rdma/link.c
> >@@ -0,0 +1,280 @@
> >+/*
> >+ * link.c	RDMA tool
> >+ *
> >+ *              This program is free software; you can redistribute it and/or
> >+ *              modify it under the terms of the GNU General Public License
> >+ *              as published by the Free Software Foundation; either version
> >+ *              2 of the License, or (at your option) any later version.
> >+ *
> >+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
> >+ */
> >+
> >+#include "rdma.h"
> >+
> >+static int link_help(struct rdma *rd)
> >+{
> >+	pr_out("Usage: %s link show [DEV/PORT_INDEX]\n", rd->filename);
> >+	return 0;
> >+}
> >+
> >+static void link_print_caps(struct nlattr **tb)
> >+{
> >+	uint64_t caps;
> >+	uint32_t idx;
> >+
> >+	/*
> >+	 * FIXME: move to indexes when kernel will start exporting them.
>
> Not exported yet?

Not yet, I want to minimize the UAPI export from kernel before user-space
part is accepted.

>
>
>
> >+	 */
> >+	static const char *link_caps[64] = {
>
> []

It will require from me to fill all 64 fields.
In current version, I'm leveraging the fact that static is initialized
to zero (NULL).

>
>
> >+		"UNKNOWN",
> >+		"SM",
> >+		"NOTICE",
> >+		"TRAP",
> >+		"OPT_IPD",
> >+		"AUTO_MIGR",
> >+		"SL_MAP",
> >+		"MKEY_NVRAM",
> >+		"PKEY_NVRAM",
> >+		"LED_INFO",
> >+		"SM_DISABLED",
> >+		"SYS_IMAGE_GUID",
> >+		"PKEY_SW_EXT_PORT_TRAP",
> >+		"UNKNOWN",
> >+		"EXTENDED_SPEEDS",
> >+		"UNKNOWN",
> >+		"CM",
> >+		"SNMP_TUNNEL",
> >+		"REINIT",
> >+		"DEVICE_MGMT",
> >+		"VENDOR_CLASS",
> >+		"DR_NOTICE",
> >+		"CAP_MASK_NOTICE",
> >+		"BOOT_MGMT",
> >+		"LINK_LATENCY",
> >+		"CLIENT_REG",
> >+		"IP_BASED_GIDS",
> >+	};
> >+
> >+	if (!tb[RDMA_NLDEV_ATTR_CAP_FLAGS])
> >+		return;
> >+
> >+	caps = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_CAP_FLAGS]);
> >+
> >+	pr_out("\n    caps: <");
> >+	for (idx = 0; idx < 64; idx++) {
> >+		if (caps & 0x1) {
> >+			pr_out("%s", link_caps[idx]?link_caps[idx]:"UNKNONW");
>
> "link_caps[idx] ? link_caps[idx] : "UNKNOWN""
>
> note the s/UNKNONW/UNKNOWN/

Right

>
>
> >+			if (caps >> 0x1)
> >+				pr_out(", ");
> >+		}
> >+		caps >>= 0x1;
>
> Interesting.
>
>
> >+	}
> >+
> >+	pr_out(">");
> >+}
> >+
> >+static void link_print_subnet_prefix(struct nlattr **tb)
> >+{
> >+	uint64_t subnet_prefix;
> >+	uint16_t sp[4];
> >+
> >+	if (!tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX])
> >+		return;
> >+
> >+	subnet_prefix = mnl_attr_get_u64(tb[RDMA_NLDEV_ATTR_SUBNET_PREFIX]);
> >+	memcpy(sp, &subnet_prefix, sizeof(uint64_t));
> >+	pr_out("subnet_prefix %04x:%04x:%04x:%04x ", sp[3], sp[2], sp[1], sp[0]);
>
> You have similar pr_out helper in the previous patch. Perhaps you can
> re-use it?

Sure

>
>
> >+}
> >+
> >+static void link_print_lid(struct nlattr **tb)
> >+{
> >+	if (!tb[RDMA_NLDEV_ATTR_LID])
> >+		return;
> >+
> >+	pr_out("lid %u ",
> >+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_LID]));
> >+}
> >+
> >+static void link_print_sm_lid(struct nlattr **tb)
> >+{
> >+
>
> Avoid the extra empty line.
>

Will remove

>
> >+	if (!tb[RDMA_NLDEV_ATTR_SM_LID])
> >+		return;
> >+
> >+	pr_out("sm_lid %u ",
> >+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_SM_LID]));
> >+}
> >+
> >+static void link_print_lmc(struct nlattr **tb)
> >+{
> >+	if (!tb[RDMA_NLDEV_ATTR_LMC])
> >+		return;
> >+
> >+	pr_out("lmc %u ", mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_LMC]));
> >+}
> >+
> >+static void link_print_state(struct nlattr **tb)
> >+{
> >+	uint8_t state;
> >+	/*
> >+	 * FIXME: move to index exported by the kernel
>
> Again, can't this be fixed now?
>

Not yet, it will be fixed in the second step.

>
> >+	 */
> >+	static const char *str[] = {
> >+		"NOP",
> >+		"DOWN",
> >+		"INIT",
> >+		"ARMED",
> >+		"ACTIVE",
> >+		"ACTIVE_DEFER",
> >+	};
> >+
> >+	if (!tb[RDMA_NLDEV_ATTR_PORT_STATE])
> >+		return;
> >+
> >+	state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_STATE]);
> >+
> >+	if (state < 6 )
>
> Magic?
>
>
> >+		pr_out("state %s ", str[state]);
> >+	else
> >+		pr_out("state UNKNOWN ");
> >+}
> >+
> >+static void link_print_phys_state(struct nlattr **tb)
> >+{
> >+	uint8_t phys_state;
> >+	/*
> >+	 * FIXME: move to index exported by the kernel
>
> Again, can't this be fixed now?
>
>
> >+	 */
> >+	static const char *str[] = {
> >+		"UNKNOWN",
> >+		"SLEEP",
> >+		"POLLING",
> >+		"DISABLED",
> >+		"PORT_CONFIGURATION_TRAINING",
> >+		"LINK_UP",
> >+		"LINK_ERROR_RECOVER",
> >+		"PHY_TEST",
> >+	};
> >+
> >+	if (!tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE])
> >+		return;
> >+
> >+	phys_state = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_PORT_PHYS_STATE]);
> >+	if (phys_state < 8)
>
> Magic?
>
>
> >+		pr_out("physical_state %s ", str[phys_state]);
> >+	else
> >+		pr_out("physical_state UNKNOWN ");
> >+}
> >+
> >+static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
> >+{
> >+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
> >+	struct rdma *rd = data;
> >+
> >+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
> >+	if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_DEV_NAME])
> >+		return MNL_CB_ERROR;
> >+
> >+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
> >+		pr_err("This tool doesn't support switches yet\n");
> >+		return MNL_CB_ERROR;
> >+	}
> >+
> >+	pr_out("%u/%u: %s/%u: ",
> >+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]),
> >+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]),
> >+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]),
> >+	       mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]));
> >+	link_print_subnet_prefix(tb);
> >+	link_print_lid(tb);
> >+	link_print_sm_lid(tb);
> >+	link_print_lmc(tb);
> >+	link_print_state(tb);
> >+	link_print_phys_state(tb);
> >+	if (rd->show_details)
> >+		link_print_caps(tb);
> >+
> >+	pr_out("\n");
> >+	return MNL_CB_OK;
> >+}
> >+
> >+static int link_no_args(struct rdma *rd)
> >+{
> >+	uint32_t seq;
> >+	int ret;
> >+
> >+	rdma_prepare_msg(rd, RDMA_NLDEV_CMD_PORT_GET, &seq, (NLM_F_REQUEST | NLM_F_ACK));
> >+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
> >+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_PORT_INDEX, rd->port_idx);
> >+	if ((ret = rdma_send_msg(rd)))
> >+		return ret;
> >+
> >+	return rdma_recv_msg(rd, link_parse_cb, rd, seq);
> >+}
> >+
> >+static int link_one_show(struct rdma *rd)
> >+{
> >+	const struct rdma_cmd cmds[] = {
> >+		{ NULL,		link_no_args},
> >+		{ 0 }
> >+	};
> >+
> >+	return rdma_exec_cmd(rd, cmds, "parameter");
> >+
>
> Avoid the extra empty line.
>
>
> >+}
> >+
> >+static int link_show(struct rdma *rd)
> >+{
> >+	struct dev_map *dev_map;
> >+	uint32_t port;
> >+	int ret = 0;
> >+
> >+	if (rd_no_arg(rd)) {
> >+		list_for_each_entry(dev_map, &rd->dev_map_list, list) {
> >+			rd->dev_idx = dev_map->idx;
> >+			for (port = 1; port < dev_map->num_ports + 1; port++) {
> >+				rd->port_idx = port;
> >+				ret = link_one_show(rd);
> >+				if (ret)
> >+					return ret;
> >+			}
> >+		}
> >+
> >+	}
> >+	else {
> >+		dev_map = dev_map_lookup(rd, true);
> >+		port = get_port_from_argv(rd);
> >+		if (!dev_map || port > dev_map->num_ports) {
> >+			pr_err("Wrong device name\n");
> >+			return -ENOENT;
> >+		}
> >+		rd_arg_inc(rd);
> >+		rd->port_idx = port ? :1;
>
> "port ? : 1"

Yeah, legal C, the same as (port) ? port : 1
ihttps://en.wikipedia.org/wiki/%3F:#C

>
> >+		for (; port < dev_map->num_ports + 1; port++, rd->port_idx++) {
> >+			ret = link_one_show(rd);
> >+			if (ret)
> >+				return ret;
> >+			if (port)
> >+				/*
> >+				 * We got request to show link for devname
> >+				 * without port index.
> >+				 */
> >+				break;
> >+		}
> >+
> >+	}
> >+	return ret;
>
> 	You can do return 0 here and avoid ret initialization.
>

Sure,

Thank you for the review.


>
> >+}
> >+
> >+int cmd_link(struct rdma *rd)
> >+{
> >+	const struct rdma_cmd cmds[] = {
> >+		{ NULL,		link_show },
> >+		{ "show",	link_show },
> >+		{ "list",	link_show },
> >+		{ "help",	link_help },
> >+		{ 0 }
> >+	};
> >+
> >+	return rdma_exec_cmd(rd, cmds, "link command");
> >+}
> >diff --git a/rdma/rdma.c b/rdma/rdma.c
> >index dfebd71e..f597f614 100644
> >--- a/rdma/rdma.c
> >+++ b/rdma/rdma.c
> >@@ -18,7 +18,7 @@
> > static void help(char *name)
> > {
> > 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
> >-	       "where  OBJECT := { dev | help }\n"
> >+	       "where  OBJECT := { dev | link | help }\n"
> > 	       "       OPTIONS := { -V[ersion] | -d[etails]}\n", name);
> > }
> >
> >@@ -34,6 +34,7 @@ static int rd_cmd(struct rdma *rd)
> > 		{ NULL,		cmd_help },
> > 		{ "help",	cmd_help },
> > 		{ "dev",	cmd_dev },
> >+		{ "link",	cmd_link },
> > 		{ 0 }
> > 	};
> >
> >diff --git a/rdma/utils.c b/rdma/utils.c
> >index bee490da..ca61ccc1 100644
> >--- a/rdma/utils.c
> >+++ b/rdma/utils.c
> >@@ -104,6 +104,11 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
> > 	[RDMA_NLDEV_ATTR_FW_VERSION] = MNL_TYPE_NUL_STRING,
> > 	[RDMA_NLDEV_ATTR_NODE_GUID] = MNL_TYPE_U64,
> > 	[RDMA_NLDEV_ATTR_SYS_IMAGE_GUID] = MNL_TYPE_U64,
> >+	[RDMA_NLDEV_ATTR_LID] = MNL_TYPE_U32,
> >+	[RDMA_NLDEV_ATTR_SM_LID] = MNL_TYPE_U32,
> >+	[RDMA_NLDEV_ATTR_LMC] = MNL_TYPE_U8,
> >+	[RDMA_NLDEV_ATTR_PORT_STATE] = MNL_TYPE_U8,
> >+	[RDMA_NLDEV_ATTR_PORT_PHYS_STATE] = MNL_TYPE_U8,
> > 	[RDMA_NLDEV_ATTR_DEV_NODE_TYPE] = MNL_TYPE_U8,
> > };
> >
> >--
> >2.13.2
> >
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 3/4] rdma: Add link object

[Jiri Pirko]

Mon, Jul 10, 2017 at 06:22:23PM CEST, leon@kernel.org wrote:
>On Mon, Jul 10, 2017 at 10:13:07AM +0200, Jiri Pirko wrote:
>> Tue, Jul 04, 2017 at 09:55:40AM CEST, leon@kernel.org wrote:
>> >From: Leon Romanovsky <leonro@mellanox.com>
>> >
>> >Link (port) object represent struct ib_port to the user space.
>> >
>> >Link properties:
>> > * Port capabilities
>> > * IB subnet prefix
>> > * LID, SM_LID and LMC
>> > * Port state
>> > * Physical state
>> >
>> >Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
>> >---
>> > rdma/Makefile |   2 +-
>> > rdma/link.c   | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > rdma/rdma.c   |   3 +-
>> > rdma/utils.c  |   5 ++
>> > 4 files changed, 288 insertions(+), 2 deletions(-)
>> > create mode 100644 rdma/link.c
>> >
>> >diff --git a/rdma/Makefile b/rdma/Makefile
>> >index 123d7ac5..1a9e4b1a 100644
>> >--- a/rdma/Makefile
>> >+++ b/rdma/Makefile
>> >@@ -2,7 +2,7 @@ include ../Config
>> >
>> > ifeq ($(HAVE_MNL),y)
>> >
>> >-RDMA_OBJ = rdma.o utils.o dev.o
>> >+RDMA_OBJ = rdma.o utils.o dev.o link.o
>> >
>> > TARGETS=rdma
>> > CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
>> >diff --git a/rdma/link.c b/rdma/link.c
>> >new file mode 100644
>> >index 00000000..f92b4cef
>> >--- /dev/null
>> >+++ b/rdma/link.c
>> >@@ -0,0 +1,280 @@
>> >+/*
>> >+ * link.c	RDMA tool
>> >+ *
>> >+ *              This program is free software; you can redistribute it and/or
>> >+ *              modify it under the terms of the GNU General Public License
>> >+ *              as published by the Free Software Foundation; either version
>> >+ *              2 of the License, or (at your option) any later version.
>> >+ *
>> >+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
>> >+ */
>> >+
>> >+#include "rdma.h"
>> >+
>> >+static int link_help(struct rdma *rd)
>> >+{
>> >+	pr_out("Usage: %s link show [DEV/PORT_INDEX]\n", rd->filename);
>> >+	return 0;
>> >+}
>> >+
>> >+static void link_print_caps(struct nlattr **tb)
>> >+{
>> >+	uint64_t caps;
>> >+	uint32_t idx;
>> >+
>> >+	/*
>> >+	 * FIXME: move to indexes when kernel will start exporting them.
>>
>> Not exported yet?
>
>Not yet, I want to minimize the UAPI export from kernel before user-space
>part is accepted.

I don't get it. If you need it in userspace, you should expose it. Why
to wait? What am I missing?

[...]


>> >+		rd->port_idx = port ? :1;
>>
>> "port ? : 1"
>
>Yeah, legal C, the same as (port) ? port : 1
>ihttps://en.wikipedia.org/wiki/%3F:#C


I was referring to the missing " ". I'm a nitpicker :)

Re: [PATCH iproute2 V3 0/4] RDMAtool

[Jiri Pirko]

Mon, Jul 10, 2017 at 06:01:44PM CEST, leon@kernel.org wrote:
>On Mon, Jul 10, 2017 at 10:02:30AM +0200, Jiri Pirko wrote:
>> Tue, Jul 04, 2017 at 09:55:37AM CEST, leon@kernel.org wrote:
>> >Hi,
>> >
>> >This is third version of series implementing the RDAMtool -  the tool
>> >to configure RDMA devices. The initial proposal was sent as RFC [1] and
>> >was based on sysfs entries as POC.
>> >
>> >The current series was rewritten completely to work with RDMA netlinks as
>> >a source of user<->kernel communications. In order to achieve that, the
>> >RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].
>> >
>> >The following is an example of various runs on my machine with 5 devices
>> >(4 in IB mode and one in Ethernet mode)
>> >
>> >### Without parameters
>> >$ rdma
>> >Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
>> >where  OBJECT := { dev | link | help }
>> >       OPTIONS := { -V[ersion] | -d[etails]}
>>
>> What about json output? You will need it sooner than later. It will
>> prevent you from a lot of headaches if you implement it right away.
>> Lesson learned...
>
>I'm planning to do it in the coming kernel cycle.

Yeah, just consider pushing it in this initial patchset. Makes sense and
saves you troubles. Up to you.

Re: [PATCH iproute2 V3 3/4] rdma: Add link object

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2822 bytes --]

On Mon, Jul 10, 2017 at 08:28:28PM +0200, Jiri Pirko wrote:
> Mon, Jul 10, 2017 at 06:22:23PM CEST, leon@kernel.org wrote:
> >On Mon, Jul 10, 2017 at 10:13:07AM +0200, Jiri Pirko wrote:
> >> Tue, Jul 04, 2017 at 09:55:40AM CEST, leon@kernel.org wrote:
> >> >From: Leon Romanovsky <leonro@mellanox.com>
> >> >
> >> >Link (port) object represent struct ib_port to the user space.
> >> >
> >> >Link properties:
> >> > * Port capabilities
> >> > * IB subnet prefix
> >> > * LID, SM_LID and LMC
> >> > * Port state
> >> > * Physical state
> >> >
> >> >Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> >> >---
> >> > rdma/Makefile |   2 +-
> >> > rdma/link.c   | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> > rdma/rdma.c   |   3 +-
> >> > rdma/utils.c  |   5 ++
> >> > 4 files changed, 288 insertions(+), 2 deletions(-)
> >> > create mode 100644 rdma/link.c
> >> >
> >> >diff --git a/rdma/Makefile b/rdma/Makefile
> >> >index 123d7ac5..1a9e4b1a 100644
> >> >--- a/rdma/Makefile
> >> >+++ b/rdma/Makefile
> >> >@@ -2,7 +2,7 @@ include ../Config
> >> >
> >> > ifeq ($(HAVE_MNL),y)
> >> >
> >> >-RDMA_OBJ = rdma.o utils.o dev.o
> >> >+RDMA_OBJ = rdma.o utils.o dev.o link.o
> >> >
> >> > TARGETS=rdma
> >> > CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
> >> >diff --git a/rdma/link.c b/rdma/link.c
> >> >new file mode 100644
> >> >index 00000000..f92b4cef
> >> >--- /dev/null
> >> >+++ b/rdma/link.c
> >> >@@ -0,0 +1,280 @@
> >> >+/*
> >> >+ * link.c	RDMA tool
> >> >+ *
> >> >+ *              This program is free software; you can redistribute it and/or
> >> >+ *              modify it under the terms of the GNU General Public License
> >> >+ *              as published by the Free Software Foundation; either version
> >> >+ *              2 of the License, or (at your option) any later version.
> >> >+ *
> >> >+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
> >> >+ */
> >> >+
> >> >+#include "rdma.h"
> >> >+
> >> >+static int link_help(struct rdma *rd)
> >> >+{
> >> >+	pr_out("Usage: %s link show [DEV/PORT_INDEX]\n", rd->filename);
> >> >+	return 0;
> >> >+}
> >> >+
> >> >+static void link_print_caps(struct nlattr **tb)
> >> >+{
> >> >+	uint64_t caps;
> >> >+	uint32_t idx;
> >> >+
> >> >+	/*
> >> >+	 * FIXME: move to indexes when kernel will start exporting them.
> >>
> >> Not exported yet?
> >
> >Not yet, I want to minimize the UAPI export from kernel before user-space
> >part is accepted.
>
> I don't get it. If you need it in userspace, you should expose it. Why
> to wait? What am I missing?

Mainly my attempt to avoid constant rebasing for four series at the
same time. One for rdmatool, one for RDMA netlink, one for RDMA UAPI changes
and one for rdma-core [1] which should reuse those exported structures too.

[1] http://github.com/linux-rdma/rdma-core

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 0/4] RDMAtool

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On Mon, Jul 10, 2017 at 09:43:25AM +0200, Jiri Pirko wrote:
> Tue, Jul 04, 2017 at 09:55:37AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
> >Hi,
> >
> >This is third version of series implementing the RDAMtool -  the tool
>
> s/RDAMtool/RDMAtool/
>
>
> >to configure RDMA devices. The initial proposal was sent as RFC [1] and
> >was based on sysfs entries as POC.
> >
> >The current series was rewritten completely to work with RDMA netlinks as
> >a source of user<->kernel communications. In order to achieve that, the
> >RDMA netlinks were extensively refactored and modernized [2, 3, 4 and 5].
> >
> >The following is an example of various runs on my machine with 5 devices
> >(4 in IB mode and one in Ethernet mode)
> >
> >### Without parameters
> >$ rdma
> >Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
> >where  OBJECT := { dev | link | help }
> >       OPTIONS := { -V[ersion] | -d[etails]}
>
> Do you plan to add "mon" any time soon?

I don't know yet.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 2/4] rdma: Add dev object

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3965 bytes --]

On Mon, Jul 10, 2017 at 10:01:02AM +0200, Jiri Pirko wrote:
> Tue, Jul 04, 2017 at 09:55:39AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
> >From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >
> >Device (dev) object represents struct ib_device to the user space.
> >
> >Device properties:
> > * Device capabilities
> > * FW version to the device output
> > * node_guid and sys_image_guid
> > * node_type
> >
> >Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >---
> > rdma/Makefile |   2 +-
> > rdma/dev.c    | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > rdma/rdma.c   |   3 +-
> > rdma/rdma.h   |  12 ++-
> > rdma/utils.c  |  46 +++++++++++-
> > 5 files changed, 293 insertions(+), 5 deletions(-)
> > create mode 100644 rdma/dev.c
> >
> >diff --git a/rdma/Makefile b/rdma/Makefile
> >index 64da2142..123d7ac5 100644
> >--- a/rdma/Makefile
> >+++ b/rdma/Makefile
> >@@ -2,7 +2,7 @@ include ../Config
> >
> > ifeq ($(HAVE_MNL),y)
> >
> >-RDMA_OBJ = rdma.o utils.o
> >+RDMA_OBJ = rdma.o utils.o dev.o
> >
> > TARGETS=rdma
> > CFLAGS += $(shell $(PKG_CONFIG) libmnl --cflags)
> >diff --git a/rdma/dev.c b/rdma/dev.c
> >new file mode 100644
> >index 00000000..b80e5288
> >--- /dev/null
> >+++ b/rdma/dev.c
> >@@ -0,0 +1,235 @@
> >+/*
> >+ * dev.c	RDMA tool
> >+ *
> >+ *              This program is free software; you can redistribute it and/or
> >+ *              modify it under the terms of the GNU General Public License
> >+ *              as published by the Free Software Foundation; either version
> >+ *              2 of the License, or (at your option) any later version.
> >+ *
> >+ * Authors:     Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >+ */
> >+
> >+#include "rdma.h"
> >+
> >+static int dev_help(struct rdma *rd)
> >+{
> >+	pr_out("Usage: %s dev show [DEV]\n", rd->filename);
> >+	return 0;
> >+}
> >+
> >+static void dev_print_caps(struct nlattr **tb)
> >+{
> >+	uint64_t caps;
> >+	uint32_t idx;
> >+
> >+	/*
> >+	 * FIXME: move to indexes when kernel will start exporting them.
> >+	 */
> >+	static const char *dev_caps[64] = {
>
> 	static const char *dev_caps[] = {

The same as for the "link" object.

>
>
> >+		"RESIZE_MAX_WR",
> >+		"BAD_PKEY_CNTR",
> >+		"BAD_QKEY_CNTR",
> >+		"RAW_MULTI",
> >+		"AUTO_PATH_MIG",
> >+		"CHANGE_PHY_PORT",
> >+		"UD_AV_PORT_ENFORCE",
> >+		"CURR_QP_STATE_MOD",
> >+		"SHUTDOWN_PORT",
> >+		"INIT_TYPE",
> >+		"PORT_ACTIVE_EVENT",
> >+		"SYS_IMAGE_GUID",
> >+		"RC_RNR_NAK_GEN",
> >+		"SRQ_RESIZE",
> >+		"N_NOTIFY_CQ",
> >+		"LOCAL_DMA_LKEY",
> >+		"RESERVED",
> >+		"MEM_WINDOW",
> >+		"UD_IP_CSUM",
> >+		"UD_TSO",
> >+		"XRC",
> >+		"MEM_MGT_EXTENSIONS",
> >+		"BLOCK_MULTICAST_LOOPBACK",
> >+		"MEM_WINDOW_TYPE_2A",
> >+		"MEM_WINDOW_TYPE_2B",
> >+		"RC_IP_CSUM",
> >+		"RAW_IP_CSUM",
> >+		"CROSS_CHANNEL",
> >+		"MANAGED_FLOW_STEERING",
> >+		"SIGNATURE_HANDOVER",
> >+		"ON_DEMAND_PAGING",
> >+		"SG_GAPS_REG",
> >+		"VIRTUAL_FUNCTION",
> >+		"RAW_SCATTER_FCS",
> >+		"RDMA_NETDEV_OPA_VNIC",
> >+	};
> >+

<...>

> >+	if (!tb[RDMA_NLDEV_ATTR_FW_VERSION])
> >+		return;
> >+
> >+	pr_out("fw %s ",
> >+	       mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_FW_VERSION]));
> >+}
> >+
> >+static void _dev_print_be64(char *name, uint64_t val)
>
> Why the initial "_"? I think it could be avoided.
>
> Wait, I'm confused. Does kernel actually exposes BE64 in U64 attribute?
> That is wrong. Please fix it while there is still a time (if it is).

It was, I fixed it in the last version of my RDMA netlink patches as a
response to Jason's comment.

The "_be64" suffix in the name is a leftover from dark ages. I'll fix it.

>
>
> >+{
> >+	uint16_t vp[4];
> >+
> >+	memcpy(vp, &val, sizeof(uint64_t));
> >+	pr_out("%s %04x:%04x:%04x:%04x ", name, vp[3], vp[2], vp[1], vp[0]);
> >+}
> >+

<...>
>
>
> >+	}
> >+	else {
>
> "} else {"
>
> Checkpatch should warn you. Did you run it?
>

Partially.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH iproute2 V3 1/4] rdma: Add basic infrastructure for RDMA tool

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

On Mon, Jul 10, 2017 at 09:47:17AM +0200, Jiri Pirko wrote:
> Tue, Jul 04, 2017 at 09:55:38AM CEST, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org wrote:
> >From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >
> >RDMA devices are cross-functional devices from one side,
> >but very tailored for the specific markets from another.
> >
> >Such diversity caused to spread of RDMA related configuration
> >across various tools, e.g. devlink, ip, ethtool, ib specific and
> >vendor specific solutions.
> >
> >This patch adds ability to fill device and port information
> >by reading RDMA netlink.
> >
> >Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>
> [...]
>
>
> >+
> >+int rd_dev_init_cb(const struct nlmsghdr *nlh, void *data)
> >+{
> >+	struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
> >+	struct dev_map *dev_map;
> >+	struct rdma *rd = data;
> >+	const char *dev_name;
> >+
> >+	mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
> >+	if (!tb[RDMA_NLDEV_ATTR_DEV_NAME] || !tb[RDMA_NLDEV_ATTR_DEV_INDEX])
> >+		return MNL_CB_ERROR;
> >+	if (!tb[RDMA_NLDEV_ATTR_PORT_INDEX]) {
> >+		pr_err("This tool doesn't support switches yet\n");
> >+		return MNL_CB_ERROR;
> >+	}
> >+
> >+	dev_name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
> >+
> >+	dev_map = dev_map_alloc(dev_name);
> >+	if (!dev_map)
> >+		/* The main function will cleanup the allocations */
> >+		return MNL_CB_ERROR;
> >+	list_add_tail(&dev_map->list, &rd->dev_map_list);
> >+
> >+	dev_map->num_ports = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
> >+	dev_map->idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
> >+
> >+	return MNL_CB_OK;
> >+}
> >+
> >+void rdma_free_devmap(struct rdma *rd)
> >+{
> >+	if(!rd)
> >+		return;
> >+	dev_map_cleanup(rd);
> >+}
> >+
> >+int rdma_exec_cmd(struct rdma *rd, const struct rdma_cmd *cmds, const char *str)
>
> Sometimes the functions are prefixed "rd_", sometimes "rdma_".
> I think it would be nice to unify this with "rd_".

Right, I'll fix.

>
>
> >+{
> >+	const struct rdma_cmd *c;
> >+
> >+	/* First argument in objs table is default variant */
> >+	if (rd_no_arg(rd))
> >+		return cmds->func(rd);
> >+
> >+	for (c = cmds + 1; c->cmd; ++c) {
> >+		if (rd_argv_match(rd, c->cmd)) {
> >+			/* Move to next argument */
> >+			rd_arg_inc(rd);
> >+			return c->func(rd);
> >+		}
> >+	}
> >+
> >+	pr_err("Unknown %s '%s'.\n", str, rd_argv(rd));
> >+	return 0;
> >+}
> >+

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]