[PATCH 00/35] RFC: add "nvme monitor" subcommand

[PATCH 00/35] RFC: add "nvme monitor" subcommand

[mwilck]

From: Martin Wilck <mwilck@suse.com>

(Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)

This patch set adds a new subcommand **nvme monitor**. In this mode,
**nvme-cli** runs continuously, monitors events (currently, uevents) relevant
for discovery, and optionally autoconnects to newly discovered subsystems.

The monitor mode is suitable to be run in a systemd service. An appropriate
unit file is provided. As such, **nvme monitor** can be used as an alternative
to the current auto-connection mechanism based on udev rules and systemd
template units. If `--autoconnect` is active, **nvme monitor** masks the
respective udev rules in order to prevent simultaneous connection attempts
from udev and itself.

This method for discovery and autodetection has some advantages over the
current udev-rule based approach:

 * By using the `--persistent` option, users can easily control whether
   persistent discovery controllers for discovered transport addresses should
   be created and monitored for AEN events. **nvme monitor** watches known
   transport addresses, creates discovery controllers as required, and re-uses
   existing ones if possible. It keeps track of persistent discovery
   controllers it created, and tears them down on exit. When run in
   `--persistent --autoconnect` mode *in the initial ramfs*, it will keep
   discovery controllers alive, so that a new instance started after switching
   root can simply re-use them.
 * In certain situations, the systemd-based approach may miss events due to
   race conditions. This can happen e.g. if an FC remote port is detected, but
   shortly after it's detection an FC relogin procedure is necessary e.g. due to
   an RSCN. In this case, an `fc_udev_device` uevent will be received on the
   first detection and handled by an `nvme connect-all` command run from
   `nvmf-connect@.service`. The connection attempt to the rport in question will
   fail with "no such device" because of the simultaneous FC
   relogin. `nvmf-connect@.service` may not terminate immediately, because it
   attempts to establish other connections listed in the Discovery Log page it
   retrieved. When the FC relogin eventually finishes, a new uevent will be
   received, and `nvmf-connect@` will be started again, but *this has no effect*
   if the previous `nvmf-connect@` service hasn't finished yet. This is the
   general semantics of systemd services, no easy workaround exists.  **nvme
   monitor** doesn't suffer from this problem. If it sees an uevent for a
   transport address for which a discovery is already running, it will queue the
   handling of this event up and restart the discovery after it's finished.
 * Resource consumption for handling uevents is lower. Instead of running an
   udev worker, executing the rules, executing `systemctl start` from the
   worker, starting a systemd service, and starting a separate **nvme-cli**
   instance, only a single `fork()` operation is necessary. Of course, on the
   back side, the monitor itself consumes resources while it's running and
   waiting for events. On my system with 8 persistent discovery controllers,
   its RSS is ~3MB. CPU consumption is zero as long as no events occur.
 * **nvme monitor** could be easily extended to handle events for non-FC
   transports.

I've tested `fc_udev_device` handling for NVMeoFC with an Ontap target, and
AEN handling for RDMA using a Linux **nvmet** target.

### Implementation notes

I've tried to change the exisiting **nvme-cli** code as little as possible
while reusing the code from `fabrics.c`. The majority of changes in the
existing code exports formerly static functions and variables, so that they
are usable from the monitor code.

The main process just waits for events using `epoll()`. When an event is
received that necessitates a new discovery, a child is forked. This makes it
possible to fill in the configuration parameters for `do_discover()` without
interfering with the main process or other discovery tasks running in
parallel. The program tracks *transport addresses* (called "connections" in
the code) rather than NVMe controllers. In `--persistent` mode, it tries to
maintain exactly one persistent discovery connection per transport address.

Using `epoll()` may look over-engineered at this stage. I hope the better
flexibility over `poll()` (in particular, the ability to add new event sources
while waiting) will simplify future extensions and improvements.

### Todo

 * Referrals are not handled perfectly yet. They will be handled by
   `do_discover()` just as it would when called from **nvme connect-all**, but
   it would be better to pass referrals back to the main process to make it
   aware of the additional discovery controller rather than using
   recursion. The main process would e.g. know if a discovery is already
   running for the transport address in the referrral.
 *  When "add" uevents for nvme controller devices are received, the
    controller is consistently not in `live` state yet, and attempting to read
    the `subsysnqn` sysfs attribute returns `(efault)`. While this should
    arguably be fixed in the kernel, it could be worked around in user space
    by using timers or polling the `state` sysfs attribute for changes.
 * Parse and handle `discovery.conf` on startup.
 * Implement support for RDMA and TCP protocols.


Martin Wilck (35):
  nvme-monitor: add new stub
  monitor: create udev socket
  monitor: initialize signal handling
  monitor: add main loop for uevent monitoring
  monitor: add uevent filters
  monitor: Create a log() macro.
  fabrics: use log() macro
  monitor: add command line options to control logging
  nvme_get_ctrl_attr(): constify "path" argument
  fabrics: export do_discover(), build_options() and config
  monitor: add option -A / --autoconnect
  monitor: add helpers for __attribute__((cleanup))
  monitor: disable nvmf-autoconnect udev rules in autoconnect mode
  monitor: implement handling of fc_udev_device
  monitor: implement handling of nvme AEN events
  monitor: reset children's signal disposition
  monitor: handle SIGCHLD for terminated child processes
  monitor: add "--persistent/-p" flag
  fabrics: use "const char *" in struct config
  fabrics: export arg_str(), parse_conn_arg(), and remove_ctrl()
  nvme-cli: add "list.h"
  conn-db: add simple connection registry
  monitor: handle restart of pending discoveries
  monitor: monitor_discovery(): try to reuse existing controllers
  monitor: read existing connections on startup
  monitor: implement starting discovery controllers on startup
  monitor: implement cleanup of created discovery controllers
  monitor: basic handling of add/remove uevents for nvme controllers
  monitor: kill running discovery tasks on exit
  monitor: add connection property options from connect-all
  completions: add completions for nvme monitor
  nvmf-autoconnect: add unit file for nvme-monitor.service
  nvme-connect-all(1): fix documentation for --quiet/-S
  nvme-monitor(1): add man page for nvme-monitor
  monitor: add option --keep/-K

 Documentation/cmds-main.txt                   |    4 +
 Documentation/nvme-connect-all.1              |    8 +-
 Documentation/nvme-connect-all.html           |   10 +-
 Documentation/nvme-connect-all.txt            |    4 +-
 Documentation/nvme-monitor.1                  |  218 ++++
 Documentation/nvme-monitor.html               | 1067 +++++++++++++++++
 Documentation/nvme-monitor.txt                |  170 +++
 Makefile                                      |   10 +
 common.h                                      |   12 +
 completions/bash-nvme-completion.sh           |    6 +-
 conn-db.c                                     |  341 ++++++
 conn-db.h                                     |  141 +++
 fabrics.c                                     |  145 +--
 fabrics.h                                     |   39 +
 list.h                                        |  365 ++++++
 log.h                                         |   44 +
 monitor.c                                     |  764 ++++++++++++
 monitor.h                                     |    6 +
 nvme-builtin.h                                |    1 +
 nvme-topology.c                               |    2 +-
 nvme.c                                        |   13 +
 nvme.h                                        |    2 +-
 nvmf-autoconnect/systemd/nvme-monitor.service |   17 +
 23 files changed, 3296 insertions(+), 93 deletions(-)
 create mode 100644 Documentation/nvme-monitor.1
 create mode 100644 Documentation/nvme-monitor.html
 create mode 100644 Documentation/nvme-monitor.txt
 create mode 100644 conn-db.c
 create mode 100644 conn-db.h
 create mode 100644 list.h
 create mode 100644 log.h
 create mode 100644 monitor.c
 create mode 100644 monitor.h
 create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service

-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 01/35] nvme-monitor: add new stub

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This stub will become the nvme AEN monitor in follow-up commits.
monitor functionality depends on libudev.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 Makefile       | 10 ++++++++++
 monitor.c      | 28 ++++++++++++++++++++++++++++
 monitor.h      |  6 ++++++
 nvme-builtin.h |  1 +
 nvme.c         | 13 +++++++++++++
 5 files changed, 58 insertions(+)
 create mode 100644 monitor.c
 create mode 100644 monitor.h

diff --git a/Makefile b/Makefile
index 3ea17b5..21c9a23 100644
--- a/Makefile
+++ b/Makefile
@@ -4,6 +4,7 @@ override CPPFLAGS += -D_GNU_SOURCE -D__CHECK_ENDIAN__
 LIBUUID = $(shell $(LD) -o /dev/null -luuid >/dev/null 2>&1; echo $$?)
 LIBHUGETLBFS = $(shell $(LD) -o /dev/null -lhugetlbfs >/dev/null 2>&1; echo $$?)
 HAVE_SYSTEMD = $(shell pkg-config --exists libsystemd  --atleast-version=242; echo $$?)
+HAVE_LIBUDEV = $(shell pkg-config --exists libudev; echo $$?)
 NVME = nvme
 INSTALL ?= install
 DESTDIR =
@@ -32,6 +33,11 @@ endif
 
 INC=-Iutil
 
+ifeq ($(HAVE_LIBUDEV),0)
+	override LDFLAGS += -ludev
+	override CFLAGS += -DHAVE_LIBUDEV
+endif
+
 ifeq ($(HAVE_SYSTEMD),0)
 	override LDFLAGS += -lsystemd
 	override CFLAGS += -DHAVE_SYSTEMD
@@ -62,6 +68,10 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \
 	nvme-lightnvm.o fabrics.o nvme-models.o plugin.o \
 	nvme-status.o nvme-filters.o nvme-topology.o
 
+ifeq ($(HAVE_LIBUDEV),0)
+	OBJS += monitor.o
+endif
+
 UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o
 
 PLUGIN_OBJS :=					\
diff --git a/monitor.c b/monitor.c
new file mode 100644
index 0000000..dae4975
--- /dev/null
+++ b/monitor.c
@@ -0,0 +1,28 @@
+/*
+ * Copyright (C) 2021 SUSE LLC
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ *
+ * This file implements a simple monitor for NVMe-related uevents.
+ */
+
+#include <errno.h>
+
+#include "nvme-status.h"
+#include "monitor.h"
+
+int aen_monitor(const char *desc, int argc, char **argv)
+{
+	/* Handle as unimplemented for now */
+	return nvme_status_to_errno(-ENOTTY, true);
+}
+
diff --git a/monitor.h b/monitor.h
new file mode 100644
index 0000000..e79d3a6
--- /dev/null
+++ b/monitor.h
@@ -0,0 +1,6 @@
+#ifndef _MONITOR_H
+#define _MONITOR_H
+
+extern int aen_monitor(const char *desc, int argc, char **argv);
+
+#endif
diff --git a/nvme-builtin.h b/nvme-builtin.h
index 59896dc..74f9a3e 100644
--- a/nvme-builtin.h
+++ b/nvme-builtin.h
@@ -80,6 +80,7 @@ COMMAND_LIST(
 	ENTRY("dir-send", "Submit a Directive Send command, return results", dir_send)
 	ENTRY("virt-mgmt", "Manage Flexible Resources between Primary and Secondary Controller ", virtual_mgmt)
 	ENTRY("rpmb", "Replay Protection Memory Block commands", rpmb_cmd)
+	ENTRY("monitor", "Monitor NVMeoF AEN events", monitor_cmd)
 );
 
 #endif
diff --git a/nvme.c b/nvme.c
index d1ef6e9..d2004a2 100644
--- a/nvme.c
+++ b/nvme.c
@@ -57,6 +57,7 @@
 
 #include "argconfig.h"
 #include "fabrics.h"
+#include "monitor.h"
 
 #define CREATE_CMD
 #include "nvme-builtin.h"
@@ -5320,6 +5321,18 @@ static int disconnect_all_cmd(int argc, char **argv, struct command *command, st
 	return fabrics_disconnect_all(desc, argc, argv);
 }
 
+static int monitor_cmd(int argc, char **argv, struct command *command, struct plugin *plugin)
+{
+#ifdef HAVE_LIBUDEV
+	const char *desc = "Monitor NVMeoF AEN events";
+
+	return aen_monitor(desc, argc, argv);
+#else
+	fprintf(stderr, "nvme-cli built without libudev doesn't support the \"monitor\" subcommand\n");
+	return EOPNOTSUPP;
+#endif
+}
+
 void register_extension(struct plugin *plugin)
 {
 	plugin->parent = &nvme;
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 02/35] monitor: create udev socket

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 3 deletions(-)

diff --git a/monitor.c b/monitor.c
index dae4975..626827c 100644
--- a/monitor.c
+++ b/monitor.c
@@ -15,14 +15,58 @@
  * This file implements a simple monitor for NVMe-related uevents.
  */
 
+#include <stddef.h>
 #include <errno.h>
+#include <libudev.h>
 
 #include "nvme-status.h"
 #include "monitor.h"
 
-int aen_monitor(const char *desc, int argc, char **argv)
+static struct udev *udev;
+
+static void cleanup_monitor(struct udev_monitor **pmon)
 {
-	/* Handle as unimplemented for now */
-	return nvme_status_to_errno(-ENOTTY, true);
+	if (*pmon) {
+		udev_monitor_unref(*pmon);
+		*pmon = NULL;
+	}
 }
 
+static int create_udev_monitor(struct udev_monitor **pmon)
+{
+	struct udev_monitor *mon __attribute((cleanup(cleanup_monitor))) = NULL;
+	int ret;
+
+	if (!udev) {
+		udev = udev_new();
+		if (!udev)
+			return -ENOMEM;
+	}
+	mon = udev_monitor_new_from_netlink(udev, "kernel");
+	if (!mon)
+		return errno ? -errno : -ENOMEM;
+
+	/*
+	 * This fails in unpriviliged mode. Use the same value as udevd.
+	 * We may able to decrease this buffer size later.
+	 */
+	(void)udev_monitor_set_receive_buffer_size(mon, 128*1024*1024);
+	ret = udev_monitor_enable_receiving(mon);
+	if (ret < 0)
+		return ret;
+	*pmon = mon;
+	mon = NULL;
+	return 0;
+}
+
+int aen_monitor(const char *desc, int argc, char **argv)
+{
+	int ret;
+	struct udev_monitor *monitor;
+
+	ret = create_udev_monitor(&monitor);
+	if (ret == 0)
+		udev_monitor_unref(monitor);
+	udev = udev_unref(udev);
+	return nvme_status_to_errno(ret, true);
+}
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 03/35] monitor: initialize signal handling

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/monitor.c b/monitor.c
index 626827c..c7f4d85 100644
--- a/monitor.c
+++ b/monitor.c
@@ -16,8 +16,10 @@
  */
 
 #include <stddef.h>
+#include <stdio.h>
 #include <errno.h>
 #include <libudev.h>
+#include <signal.h>
 
 #include "nvme-status.h"
 #include "monitor.h"
@@ -59,14 +61,46 @@ static int create_udev_monitor(struct udev_monitor **pmon)
 	return 0;
 }
 
+static sig_atomic_t must_exit;
+
+static void monitor_int_handler(int sig)
+{
+	must_exit = 1;
+}
+
+static int monitor_init_signals(void)
+{
+	sigset_t mask;
+	struct sigaction sa = { .sa_handler = monitor_int_handler, };
+
+	/*
+	 * Block all signals. They will be unblocked when we wait
+	 * for events.
+	 */
+	sigfillset(&mask);
+	if (sigprocmask(SIG_BLOCK, &mask, NULL) == -1)
+		return -errno;
+	if (sigaction(SIGTERM, &sa, NULL) == -1)
+		return -errno;
+	if (sigaction(SIGINT, &sa, NULL) == -1)
+		return -errno;
+	return 0;
+}
+
 int aen_monitor(const char *desc, int argc, char **argv)
 {
 	int ret;
 	struct udev_monitor *monitor;
 
+	ret = monitor_init_signals();
+	if (ret != 0) {
+		fprintf(stderr, "monitor: failed to initialize signals: %m\n");
+		goto out;
+	}
 	ret = create_udev_monitor(&monitor);
 	if (ret == 0)
 		udev_monitor_unref(monitor);
 	udev = udev_unref(udev);
+out:
 	return nvme_status_to_errno(ret, true);
 }
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 04/35] monitor: add main loop for uevent monitoring

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 77 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index c7f4d85..63e26bc 100644
--- a/monitor.c
+++ b/monitor.c
@@ -17,15 +17,25 @@
 
 #include <stddef.h>
 #include <stdio.h>
+#include <unistd.h>
 #include <errno.h>
 #include <libudev.h>
 #include <signal.h>
+#include <sys/epoll.h>
 
 #include "nvme-status.h"
 #include "monitor.h"
 
 static struct udev *udev;
 
+static void close_ptr(int *p)
+{
+	if (*p != -1) {
+		close(*p);
+		*p = -1;
+	}
+}
+
 static void cleanup_monitor(struct udev_monitor **pmon)
 {
 	if (*pmon) {
@@ -87,6 +97,70 @@ static int monitor_init_signals(void)
 	return 0;
 }
 
+static void monitor_handle_udevice(struct udev_device *ud)
+{
+	fprintf(stderr, "uevent: %s %s\n",
+		udev_device_get_action(ud),
+		udev_device_get_sysname(ud));
+}
+
+static void monitor_handle_uevents(struct udev_monitor *monitor)
+{
+	struct udev_device *ud;
+
+	for (ud = udev_monitor_receive_device(monitor);
+	     ud;
+	     ud = udev_monitor_receive_device(monitor)) {
+		monitor_handle_udevice(ud);
+		udev_device_unref(ud);
+	}
+}
+
+#define MAX_EVENTS 1
+static int monitor_main_loop(struct udev_monitor *monitor)
+{
+	int ep_fd __attribute__((cleanup(close_ptr))) = -1;
+	int ret;
+	struct epoll_event ep_ev = { .events = EPOLLIN, };
+	struct epoll_event events[MAX_EVENTS];
+	sigset_t ep_mask;
+
+	ep_fd = epoll_create1(0);
+	if (ep_fd == -1)
+		return -errno;
+	ep_ev.data.ptr = monitor;
+	ret = epoll_ctl(ep_fd, EPOLL_CTL_ADD,
+			udev_monitor_get_fd(monitor), &ep_ev);
+	if (ret == -1)
+		return -errno;
+
+	sigfillset(&ep_mask);
+	sigdelset(&ep_mask, SIGTERM);
+	sigdelset(&ep_mask, SIGINT);
+	while (1) {
+		int rc, i;
+
+		rc = epoll_pwait(ep_fd, events, MAX_EVENTS, -1, &ep_mask);
+		if (rc == -1 && errno == EINTR) {
+			fprintf(stderr, "monitor: exit signal received\n");
+			return 0;
+		} else if (rc == -1) {
+			fprintf(stderr, "monitor: epoll_wait: %m\n");
+			return -errno;
+		} else if (rc == 0 || rc > MAX_EVENTS) {
+			fprintf(stderr, "monitor: epoll_wait: unexpected rc=%d\n", rc);
+			continue;
+		}
+		for (i = 0; i < MAX_EVENTS; i++) {
+			if (events[i].data.ptr == monitor)
+				(void)monitor_handle_uevents(monitor);
+			else
+				fprintf(stderr, "monitor: unexpected event\n");
+		}
+	}
+	return ret;
+}
+
 int aen_monitor(const char *desc, int argc, char **argv)
 {
 	int ret;
@@ -98,8 +172,10 @@ int aen_monitor(const char *desc, int argc, char **argv)
 		goto out;
 	}
 	ret = create_udev_monitor(&monitor);
-	if (ret == 0)
+	if (ret == 0) {
+		ret = monitor_main_loop(monitor);
 		udev_monitor_unref(monitor);
+	}
 	udev = udev_unref(udev);
 out:
 	return nvme_status_to_errno(ret, true);
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 05/35] monitor: add uevent filters

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/monitor.c b/monitor.c
index 63e26bc..a6c8905 100644
--- a/monitor.c
+++ b/monitor.c
@@ -58,6 +58,10 @@ static int create_udev_monitor(struct udev_monitor **pmon)
 	if (!mon)
 		return errno ? -errno : -ENOMEM;
 
+	/* Add match for NVMe controller devices */
+	ret = udev_monitor_filter_add_match_subsystem_devtype(mon, "nvme", NULL);
+	/* Add match for fc_udev_device */
+	ret = udev_monitor_filter_add_match_subsystem_devtype(mon, "fc", NULL);
 	/*
 	 * This fails in unpriviliged mode. Use the same value as udevd.
 	 * We may able to decrease this buffer size later.
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 06/35] monitor: Create a log() macro.

[mwilck]

From: Martin Wilck <mwilck@suse.com>

As this is a long running program, we need to make the log output
configurable. First step: replace fprintf() by log(). The log level
and printing of time stamps can be configured at run time using
global variables. These will live in fabrics.c.

Allow toggling function name printing at build time.
Printing the function name is useful for development, but perhaps
not desired for production.
Put '#define LOG_FUNCNAME' before '#include "log.h"' to switch it on.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 log.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 log.h

diff --git a/log.h b/log.h
new file mode 100644
index 0000000..2017731
--- /dev/null
+++ b/log.h
@@ -0,0 +1,44 @@
+#ifndef _LOG_H
+#define _LOG_H
+
+#ifndef MAX_LOGLEVEL
+#  define MAX_LOGLEVEL LOG_DEBUG
+#endif
+#ifndef DEFAULT_LOGLEVEL
+#  define DEFAULT_LOGLEVEL LOG_NOTICE
+#endif
+
+#ifdef LOG_FUNCNAME
+#define _func_fmt "%s: "
+#define _func_arg __func__
+#else
+#define _func_fmt "%s"
+#define _func_arg ""
+#endif
+
+extern int log_level;
+extern bool log_timestamp;
+#define _TIME_FMT "[%ld.%06ld] "
+#define log(lvl, format, ...) \
+	do {								\
+		int __lvl = (lvl);					\
+									\
+		if (__lvl <= MAX_LOGLEVEL && __lvl <= log_level) {	\
+			if (log_timestamp) {				\
+				struct timespec __ts;			\
+									\
+				clock_gettime(CLOCK_MONOTONIC, &__ts);	\
+				fprintf(stderr,				\
+					_TIME_FMT _func_fmt format,	\
+					__ts.tv_sec, __ts.tv_nsec / 1000,\
+					_func_arg,			\
+					##__VA_ARGS__);			\
+			} else {					\
+				fprintf(stderr, _func_fmt format,	\
+					_func_arg,			\
+					##__VA_ARGS__);			\
+			};						\
+		}							\
+	} while (0)
+
+#endif /* _LOG_H */
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 07/35] fabrics: use log() macro

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Fabrics functionality will be used by the monitor code, we
need consistent logging. This patch causes no functional change
for the fabrics module.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 fabrics.c | 83 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 45 insertions(+), 38 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index 17711cf..555b6b4 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -32,6 +32,8 @@
 #include <libgen.h>
 #include <sys/stat.h>
 #include <stddef.h>
+#include <syslog.h>
+#include <time.h>
 
 #include <sys/types.h>
 #include <arpa/inet.h>
@@ -46,6 +48,10 @@
 #include "util/argconfig.h"
 
 #include "common.h"
+#include "log.h"
+
+int log_level = LOG_NOTICE;
+bool log_timestamp;
 
 #ifdef HAVE_SYSTEMD
 #include <systemd/sd-id128.h>
@@ -86,7 +92,6 @@ static struct config {
 	int  hdr_digest;
 	int  data_digest;
 	bool persistent;
-	bool quiet;
 	bool matching_only;
 } cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
 
@@ -345,7 +350,7 @@ static char *find_ctrl_with_connectargs(struct connect_args *args)
 
 	n = scandir(SYS_NVME, &devices, scan_ctrls_filter, alphasort);
 	if (n < 0) {
-		fprintf(stderr, "no NVMe controller(s) detected.\n");
+		log(LOG_ERR, "no NVMe controller(s) detected.\n");
 		return NULL;
 	}
 
@@ -353,7 +358,7 @@ static char *find_ctrl_with_connectargs(struct connect_args *args)
 		if (ctrl_matches_connectargs(devices[i]->d_name, args)) {
 			devname = strdup(devices[i]->d_name);
 			if (devname == NULL)
-				fprintf(stderr, "no memory for ctrl name %s\n",
+				log(LOG_ERR, "no memory for ctrl name %s\n",
 						devices[i]->d_name);
 			goto cleanup_devices;
 		}
@@ -415,7 +420,7 @@ static int add_ctrl(const char *argstr)
 
 	fd = open(PATH_NVME_FABRICS, O_RDWR);
 	if (fd < 0) {
-		fprintf(stderr, "Failed to open %s: %s\n",
+		log(LOG_ERR, "Failed to open %s: %s\n",
 			 PATH_NVME_FABRICS, strerror(errno));
 		ret = -errno;
 		goto out;
@@ -423,8 +428,8 @@ static int add_ctrl(const char *argstr)
 
 	ret = write(fd, argstr, len);
 	if (ret != len) {
-		if (errno != EALREADY || !cfg.quiet)
-			fprintf(stderr, "Failed to write to %s: %s\n",
+		if (errno != EALREADY)
+			log(LOG_NOTICE, "Failed to write to %s: %s\n",
 				 PATH_NVME_FABRICS, strerror(errno));
 		ret = -errno;
 		goto out_close;
@@ -432,7 +437,7 @@ static int add_ctrl(const char *argstr)
 
 	len = read(fd, buf, BUF_SIZE);
 	if (len < 0) {
-		fprintf(stderr, "Failed to read from %s: %s\n",
+		log(LOG_ERR, "Failed to read from %s: %s\n",
 			 PATH_NVME_FABRICS, strerror(errno));
 		ret = -errno;
 		goto out_close;
@@ -459,7 +464,7 @@ static int add_ctrl(const char *argstr)
 	}
 
 out_fail:
-	fprintf(stderr, "Failed to parse ctrl info for \"%s\"\n", argstr);
+	log(LOG_ERR, "Failed to parse ctrl info for \"%s\"\n", argstr);
 	ret = -EINVAL;
 out_close:
 	close(fd);
@@ -474,7 +479,7 @@ static int remove_ctrl_by_path(char *sysfs_path)
 	fd = open(sysfs_path, O_WRONLY);
 	if (fd < 0) {
 		ret = -errno;
-		fprintf(stderr, "Failed to open %s: %s\n", sysfs_path,
+		log(LOG_ERR, "Failed to open %s: %s\n", sysfs_path,
 				strerror(errno));
 		goto out;
 	}
@@ -528,7 +533,7 @@ static int nvmf_get_log_page_discovery(const char *dev_path,
 	fd = open(dev_path, O_RDWR);
 	if (fd < 0) {
 		error = -errno;
-		fprintf(stderr, "Failed to open %s: %s\n",
+		log(LOG_ERR, "Failed to open %s: %s\n",
 				dev_path, strerror(errno));
 		goto out;
 	}
@@ -699,7 +704,7 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec)
 
 	fd = open(cfg.raw, O_CREAT|O_RDWR|O_TRUNC, S_IRUSR|S_IWUSR);
 	if (fd < 0) {
-		fprintf(stderr, "failed to open %s: %s\n",
+		log(LOG_ERR, "failed to open %s: %s\n",
 			cfg.raw, strerror(errno));
 		return;
 	}
@@ -708,7 +713,7 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec)
 			numrec * sizeof(struct nvmf_disc_rsp_page_entry);
 	ret = write(fd, log, len);
 	if (ret < 0)
-		fprintf(stderr, "failed to write to %s: %s\n",
+		log(LOG_ERR, "failed to write to %s: %s\n",
 			cfg.raw, strerror(errno));
 	else
 		printf("Discovery log is saved to %s\n", cfg.raw);
@@ -855,13 +860,13 @@ static int build_options(char *argstr, int max_len, bool discover)
 	int len;
 
 	if (!cfg.transport) {
-		fprintf(stderr, "need a transport (-t) argument\n");
+		log(LOG_ERR, "need a transport (-t) argument\n");
 		return -EINVAL;
 	}
 
 	if (strncmp(cfg.transport, "loop", 4)) {
 		if (!cfg.traddr) {
-			fprintf(stderr, "need a address (-a) argument\n");
+			log(LOG_ERR, "need a address (-a) argument\n");
 			return -EINVAL;
 		}
 	}
@@ -945,7 +950,7 @@ static int hostname2traddr(struct config *cfg)
 
 	ret = getaddrinfo(cfg->traddr, NULL, &hints, &host_info);
 	if (ret) {
-		fprintf(stderr, "failed to resolve host %s info\n", cfg->traddr);
+		log(LOG_ERR, "failed to resolve host %s info\n", cfg->traddr);
 		return ret;
 	}
 
@@ -961,14 +966,14 @@ static int hostname2traddr(struct config *cfg)
 			addrstr, NVMF_TRADDR_SIZE);
 		break;
 	default:
-		fprintf(stderr, "unrecognized address family (%d) %s\n",
+		log(LOG_ERR, "unrecognized address family (%d) %s\n",
 			host_info->ai_family, cfg->traddr);
 		ret = -EINVAL;
 		goto free_addrinfo;
 	}
 
 	if (!p) {
-		fprintf(stderr, "failed to get traddr for %s\n", cfg->traddr);
+		log(LOG_ERR, "failed to get traddr for %s\n", cfg->traddr);
 		ret = -errno;
 		goto free_addrinfo;
 	}
@@ -996,7 +1001,7 @@ retry:
 	case NVME_NQN_NVME:
 		break;
 	default:
-		fprintf(stderr, "skipping unsupported subtype %d\n",
+		log(LOG_ERR, "skipping unsupported subtype %d\n",
 			 e->subtype);
 		return -EINVAL;
 	}
@@ -1085,7 +1090,7 @@ retry:
 
 	transport = trtype_str(e->trtype);
 	if (!strcmp(transport, "unrecognized")) {
-		fprintf(stderr, "skipping unsupported transport %d\n",
+		log(LOG_ERR, "skipping unsupported transport %d\n",
 				 e->trtype);
 		return -EINVAL;
 	}
@@ -1131,7 +1136,7 @@ retry:
 			p += len;
 			break;
 		default:
-			fprintf(stderr, "skipping unsupported adrfam\n");
+			log(LOG_ERR, "skipping unsupported adrfam\n");
 			return -EINVAL;
 		}
 		break;
@@ -1146,7 +1151,7 @@ retry:
 			p += len;
 			break;
 		default:
-			fprintf(stderr, "skipping unsupported adrfam\n");
+			log(LOG_ERR, "skipping unsupported adrfam\n");
 			return -EINVAL;
 		}
 		break;
@@ -1231,9 +1236,7 @@ static int connect_ctrls(struct nvmf_disc_rsp_page_hdr *log, int numrec)
 		if (instance == -EALREADY) {
 			const char *traddr = log->entries[i].traddr;
 
-			if (!cfg.quiet)
-				fprintf(stderr,
-					"traddr=%.*s is already connected\n",
+			log(LOG_NOTICE, "traddr=%.*s is already connected\n",
 					space_strip_len(NVMF_TRADDR_SIZE,
 							traddr),
 					traddr);
@@ -1321,12 +1324,12 @@ static int do_discover(char *argstr, bool connect)
 			print_discovery_log(log, numrec);
 		break;
 	case DISC_GET_NUMRECS:
-		fprintf(stderr,
+		log(LOG_ERR,
 			"Get number of discovery log entries failed.\n");
 		ret = status;
 		break;
 	case DISC_GET_LOG:
-		fprintf(stderr, "Get discovery log entries failed.\n");
+		log(LOG_ERR, "Get discovery log entries failed.\n");
 		ret = status;
 		break;
 	case DISC_NO_LOG:
@@ -1338,12 +1341,12 @@ static int do_discover(char *argstr, bool connect)
 		ret = -EAGAIN;
 		break;
 	case DISC_NOT_EQUAL:
-		fprintf(stderr,
+		log(LOG_ERR,
 		"Numrec values of last two get discovery log page not equal\n");
 		ret = -EBADSLT;
 		break;
 	default:
-		fprintf(stderr, "Get discovery log page failed: %d\n", ret);
+		log(LOG_ERR, "Get discovery log page failed: %d\n", ret);
 		break;
 	}
 
@@ -1359,7 +1362,7 @@ static int discover_from_conf_file(const char *desc, char *argstr,
 
 	f = fopen(PATH_NVMF_DISC, "r");
 	if (f == NULL) {
-		fprintf(stderr, "No discover params given and no %s conf\n",
+		log(LOG_ERR, "No discover params given and no %s conf\n",
 			PATH_NVMF_DISC);
 		return -EINVAL;
 	}
@@ -1370,14 +1373,14 @@ static int discover_from_conf_file(const char *desc, char *argstr,
 
 		args = strdup(line);
 		if (!args) {
-			fprintf(stderr, "failed to strdup args\n");
+			log(LOG_ERR, "failed to strdup args\n");
 			ret = -ENOMEM;
 			goto out;
 		}
 
 		argv = calloc(MAX_DISC_ARGS, BUF_SIZE);
 		if (!argv) {
-			fprintf(stderr, "failed to allocate argv vector\n");
+			log(LOG_ERR, "failed to allocate argv vector\n");
 			free(args);
 			ret = -ENOMEM;
 			goto out;
@@ -1428,6 +1431,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)
 {
 	char argstr[BUF_SIZE];
 	int ret;
+	bool quiet = false;
 
 	OPT_ARGS(opts) = {
 		OPT_LIST("transport",      't', &cfg.transport,       "transport type"),
@@ -1449,7 +1453,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)
 		OPT_INT("nr-poll-queues",  'P', &cfg.nr_poll_queues,  "number of poll queues to use (default 0)"),
 		OPT_INT("queue-size",      'Q', &cfg.queue_size,      "number of io queue elements to use (default 128)"),
 		OPT_FLAG("persistent",     'p', &cfg.persistent,      "persistent discovery connection"),
-		OPT_FLAG("quiet",          'S', &cfg.quiet,           "suppress already connected errors"),
+		OPT_FLAG("quiet",          'S', &quiet,               "suppress already connected errors"),
 		OPT_FLAG("matching",       'm', &cfg.matching_only,   "connect only records matching the traddr"),
 		OPT_END()
 	};
@@ -1459,6 +1463,9 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)
 	if (ret)
 		goto out;
 
+	if (quiet)
+		log_level = LOG_WARNING;
+
 	if (cfg.device && !strcmp(cfg.device, "none"))
 		cfg.device = NULL;
 
@@ -1534,7 +1541,7 @@ int fabrics_connect(const char *desc, int argc, char **argv)
 		goto out;
 
 	if (!cfg.nqn) {
-		fprintf(stderr, "need a -n argument\n");
+		log(LOG_ERR, "need a -n argument\n");
 		ret = -EINVAL;
 		goto out;
 	}
@@ -1572,7 +1579,7 @@ static int disconnect_subsys(char *nqn, char *ctrl)
 
 	fd = open(sysfs_nqn_path, O_RDONLY);
 	if (fd < 0) {
-		fprintf(stderr, "Failed to open %s: %s\n",
+		log(LOG_ERR, "Failed to open %s: %s\n",
 				sysfs_nqn_path, strerror(errno));
 		goto free;
 	}
@@ -1646,7 +1653,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)
 		goto out;
 
 	if (!cfg.nqn && !cfg.device) {
-		fprintf(stderr, "need a -n or -d argument\n");
+		log(LOG_ERR, "need a -n or -d argument\n");
 		ret = -EINVAL;
 		goto out;
 	}
@@ -1654,7 +1661,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)
 	if (cfg.nqn) {
 		ret = disconnect_by_nqn(cfg.nqn);
 		if (ret < 0)
-			fprintf(stderr, "Failed to disconnect by NQN: %s\n",
+			log(LOG_ERR, "Failed to disconnect by NQN: %s\n",
 				cfg.nqn);
 		else {
 			printf("NQN:%s disconnected %d controller(s)\n", cfg.nqn, ret);
@@ -1665,7 +1672,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)
 	if (cfg.device) {
 		ret = disconnect_by_device(cfg.device);
 		if (ret)
-			fprintf(stderr,
+			log(LOG_ERR,
 				"Failed to disconnect by device name: %s\n",
 				cfg.device);
 	}
@@ -1689,7 +1696,7 @@ int fabrics_disconnect_all(const char *desc, int argc, char **argv)
 
 	err = scan_subsystems(&t, NULL, 0, NULL);
 	if (err) {
-		fprintf(stderr, "Failed to scan namespaces\n");
+		log(LOG_ERR, "Failed to scan namespaces\n");
 		goto out;
 	}
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 08/35] monitor: add command line options to control logging

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Standard syslog loglevels are used.
Options:

 -S/--silent   to suppress LOG_NOTICE
 -v/--verbose  to enable LOG_INFO (overrides -S)
 -D/--debug    to enable LOG_DEBUG (overrides -S, -v)
 -C/--clockstamps to enable time stamps on log messages.

I tried to use short options that don't conflict with options
from nvme connect-all, because many of those options might be
useful for the monitor, too.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 47 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 41 insertions(+), 6 deletions(-)

diff --git a/monitor.c b/monitor.c
index a6c8905..a7afa1c 100644
--- a/monitor.c
+++ b/monitor.c
@@ -21,10 +21,15 @@
 #include <errno.h>
 #include <libudev.h>
 #include <signal.h>
+#include <time.h>
+#include <syslog.h>
 #include <sys/epoll.h>
 
 #include "nvme-status.h"
+#include "util/argconfig.h"
 #include "monitor.h"
+#define LOG_FUNCNAME 1
+#include "log.h"
 
 static struct udev *udev;
 
@@ -103,7 +108,7 @@ static int monitor_init_signals(void)
 
 static void monitor_handle_udevice(struct udev_device *ud)
 {
-	fprintf(stderr, "uevent: %s %s\n",
+	log(LOG_INFO, "uevent: %s %s\n",
 		udev_device_get_action(ud),
 		udev_device_get_sysname(ud));
 }
@@ -146,33 +151,63 @@ static int monitor_main_loop(struct udev_monitor *monitor)
 
 		rc = epoll_pwait(ep_fd, events, MAX_EVENTS, -1, &ep_mask);
 		if (rc == -1 && errno == EINTR) {
-			fprintf(stderr, "monitor: exit signal received\n");
+			log(LOG_NOTICE, "monitor: exit signal received\n");
 			return 0;
 		} else if (rc == -1) {
-			fprintf(stderr, "monitor: epoll_wait: %m\n");
+			log(LOG_ERR, "monitor: epoll_wait: %m\n");
 			return -errno;
 		} else if (rc == 0 || rc > MAX_EVENTS) {
-			fprintf(stderr, "monitor: epoll_wait: unexpected rc=%d\n", rc);
+			log(LOG_ERR, "monitor: epoll_wait: unexpected rc=%d\n", rc);
 			continue;
 		}
 		for (i = 0; i < MAX_EVENTS; i++) {
 			if (events[i].data.ptr == monitor)
 				(void)monitor_handle_uevents(monitor);
 			else
-				fprintf(stderr, "monitor: unexpected event\n");
+				log(LOG_ERR, "monitor: unexpected event\n");
 		}
 	}
 	return ret;
 }
 
+static int monitor_parse_opts(const char *desc, int argc, char **argv)
+{
+	bool quiet = false;
+	bool verbose = false;
+	bool debug = false;
+	int ret;
+	OPT_ARGS(opts) = {
+		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
+		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
+		OPT_FLAG("debug",          'D', &debug,               "log level: debug"),
+		OPT_FLAG("clockstamps",    'C', &log_timestamp,       "print log timestamps"),
+		OPT_END()
+	};
+
+	ret = argconfig_parse(argc, argv, desc, opts);
+	if (ret)
+		return ret;
+	if (quiet)
+		log_level = LOG_WARNING;
+	if (verbose)
+		log_level = LOG_INFO;
+	if (debug)
+		log_level = LOG_DEBUG;
+
+	return ret;
+}
+
 int aen_monitor(const char *desc, int argc, char **argv)
 {
 	int ret;
 	struct udev_monitor *monitor;
 
+	ret = monitor_parse_opts(desc, argc, argv);
+	if (ret)
+		goto out;
 	ret = monitor_init_signals();
 	if (ret != 0) {
-		fprintf(stderr, "monitor: failed to initialize signals: %m\n");
+		log(LOG_ERR, "monitor: failed to initialize signals: %m\n");
 		goto out;
 	}
 	ret = create_udev_monitor(&monitor);
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 09/35] nvme_get_ctrl_attr(): constify "path" argument

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 nvme-topology.c | 2 +-
 nvme.h          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/nvme-topology.c b/nvme-topology.c
index 71371c5..f62446b 100644
--- a/nvme-topology.c
+++ b/nvme-topology.c
@@ -45,7 +45,7 @@ close_fd:
 	return subsysnqn;
 }
 
-char *nvme_get_ctrl_attr(char *path, const char *attr)
+char *nvme_get_ctrl_attr(const char *path, const char *attr)
 {
 	char *attrpath, *value;
 	ssize_t ret;
diff --git a/nvme.h b/nvme.h
index 3fb1060..382e625 100644
--- a/nvme.h
+++ b/nvme.h
@@ -109,7 +109,7 @@ int scan_subsystems(struct nvme_topology *t, const char *subsysnqn,
 		    __u32 ns_instance, char *dev_dir);
 void free_topology(struct nvme_topology *t);
 char *get_nvme_subsnqn(char *path);
-char *nvme_get_ctrl_attr(char *path, const char *attr);
+char *nvme_get_ctrl_attr(const char *path, const char *attr);
 
 void *nvme_alloc(size_t len, bool *huge);
 void nvme_free(void *p, bool huge);
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 10/35] fabrics: export do_discover(), build_options() and config

[mwilck]

From: Martin Wilck <mwilck@suse.com>

These functions will be called by the monitor code.
Also export BUF_SIZE as the size of argstr.

Being able to access struct config and the "cfg" variable
from fabrics.c is essential for the monitor to leverage the existing,
well tested code as much as possible.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 fabrics.c | 33 ++++-----------------------------
 fabrics.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 29 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index 555b6b4..c1a4bb5 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -69,31 +69,7 @@ const char *conarg_traddr = "traddr";
 const char *conarg_trsvcid = "trsvcid";
 const char *conarg_host_traddr = "host_traddr";
 
-static struct config {
-	char *nqn;
-	char *transport;
-	char *traddr;
-	char *trsvcid;
-	char *host_traddr;
-	char *hostnqn;
-	char *hostid;
-	int  nr_io_queues;
-	int  nr_write_queues;
-	int  nr_poll_queues;
-	int  queue_size;
-	int  keep_alive_tmo;
-	int  reconnect_delay;
-	int  ctrl_loss_tmo;
-	int  tos;
-	char *raw;
-	char *device;
-	int  duplicate_connect;
-	int  disable_sqflow;
-	int  hdr_digest;
-	int  data_digest;
-	bool persistent;
-	bool matching_only;
-} cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
+struct config cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
 
 struct connect_args {
 	char *subsysnqn;
@@ -107,7 +83,6 @@ struct connect_args {
 
 struct connect_args *tracked_ctrls;
 
-#define BUF_SIZE		4096
 #define PATH_NVME_FABRICS	"/dev/nvme-fabrics"
 #define PATH_NVMF_DISC		"/etc/nvme/discovery.conf"
 #define PATH_NVMF_HOSTNQN	"/etc/nvme/hostnqn"
@@ -226,7 +201,7 @@ static const char *cms_str(__u8 cm)
 	return arg_str(cms, ARRAY_SIZE(cms), cm);
 }
 
-static int do_discover(char *argstr, bool connect);
+int do_discover(char *argstr, bool connect);
 
 /*
  * parse strings with connect arguments to find a particular field.
@@ -855,7 +830,7 @@ add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
 	return 0;
 }
 
-static int build_options(char *argstr, int max_len, bool discover)
+int build_options(char *argstr, int max_len, bool discover)
 {
 	int len;
 
@@ -1264,7 +1239,7 @@ static void nvmf_get_host_identifiers(int ctrl_instance)
 	cfg.hostid = nvme_get_ctrl_attr(path, "hostid");
 }
 
-static int do_discover(char *argstr, bool connect)
+int do_discover(char *argstr, bool connect)
 {
 	struct nvmf_disc_rsp_page_hdr *log = NULL;
 	char *dev_name;
diff --git a/fabrics.h b/fabrics.h
index f5b8eaf..ce965a3 100644
--- a/fabrics.h
+++ b/fabrics.h
@@ -10,4 +10,38 @@ extern int fabrics_connect(const char *desc, int argc, char **argv);
 extern int fabrics_disconnect(const char *desc, int argc, char **argv);
 extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);
 
+/* Symbols used by monitor.c */
+
+struct config {
+	char *nqn;
+	char *transport;
+	char *traddr;
+	char *trsvcid;
+	char *host_traddr;
+	char *hostnqn;
+	char *hostid;
+	int  nr_io_queues;
+	int  nr_write_queues;
+	int  nr_poll_queues;
+	int  queue_size;
+	int  keep_alive_tmo;
+	int  reconnect_delay;
+	int  ctrl_loss_tmo;
+	int  tos;
+	char *raw;
+	char *device;
+	int  duplicate_connect;
+	int  disable_sqflow;
+	int  hdr_digest;
+	int  data_digest;
+	bool persistent;
+	bool matching_only;
+};
+extern struct config cfg;
+
+#define BUF_SIZE 4096
+
+int build_options(char *argstr, int max_len, bool discover);
+int do_discover(char *argstr, bool connect);
+
 #endif
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 11/35] monitor: add option -A / --autoconnect

[mwilck]

From: Martin Wilck <mwilck@suse.com>

With this option, the monitor will try to connect to newly discovered
controllers.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/monitor.c b/monitor.c
index a7afa1c..ecf3be2 100644
--- a/monitor.c
+++ b/monitor.c
@@ -31,6 +31,10 @@
 #define LOG_FUNCNAME 1
 #include "log.h"
 
+static struct monitor_config {
+	bool autoconnect;
+} mon_cfg;
+
 static struct udev *udev;
 
 static void close_ptr(int *p)
@@ -177,6 +181,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 	bool debug = false;
 	int ret;
 	OPT_ARGS(opts) = {
+		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
 		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
 		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
 		OPT_FLAG("debug",          'D', &debug,               "log level: debug"),
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 12/35] monitor: add helpers for __attribute__((cleanup))

[mwilck]

From: Martin Wilck <mwilck@suse.com>

__attribute__((cleanup)) is very helpful but ugly. Try to avoid
defining lots of cleanup functions with these macros.

Usage: to declare an auto-cleanup variable of type (some_type *),
write

CLEANUP_FUNC(some_type)

void some_func(void)
{
        CLEANUP(some_type, varname) = NULL;
	...
}

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 common.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/common.h b/common.h
index 1c214a4..9386ec5 100644
--- a/common.h
+++ b/common.h
@@ -12,4 +12,16 @@
 #define __stringify_1(x...) #x
 #define __stringify(x...)  __stringify_1(x)
 
+#define CLEANUP_FUNC(type) \
+static void __cleanup_ ## type ##_p(type ** __p) \
+{						 \
+	if (*__p) {				 \
+		free(*__p);			 \
+		*__p = NULL;			 \
+	}					 \
+}
+
+#define CLEANUP(__t, __v) \
+	__t *__v __attribute__((cleanup(__cleanup_ ## __t ## _p)))
+
 #endif
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[mwilck]

From: Martin Wilck <mwilck@suse.com>

If autoconnect is enabled, disable the respective udev rules
by symlinking /run/udev/rules.d to /dev/null, in order to avoid
the connections being set up by the monitor and the udev workers
at the same time. This is probably the preferred mode of operation
for the monitor.

Users can override this by copying 70-nvmf-autoconnect.rules
from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
takes precedence over /run/udev/rules.d).

If the symlink can't be created for some reason, autoconnect will
be disabled. There is  only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rules already points to
/dev/null at startup, autoconnect can be left on, but the symlink
isn't removed on exit.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index ecf3be2..2a906db 100644
--- a/monitor.c
+++ b/monitor.c
@@ -17,14 +17,18 @@
 
 #include <stddef.h>
 #include <stdio.h>
+#include <stdlib.h>
 #include <unistd.h>
 #include <errno.h>
 #include <libudev.h>
 #include <signal.h>
 #include <time.h>
+#include <limits.h>
 #include <syslog.h>
+#include <sys/stat.h>
 #include <sys/epoll.h>
 
+#include "common.h"
 #include "nvme-status.h"
 #include "util/argconfig.h"
 #include "monitor.h"
@@ -33,6 +37,7 @@
 
 static struct monitor_config {
 	bool autoconnect;
+	bool skip_udev_on_exit;
 } mon_cfg;
 
 static struct udev *udev;
@@ -45,6 +50,8 @@ static void close_ptr(int *p)
 	}
 }
 
+CLEANUP_FUNC(char)
+
 static void cleanup_monitor(struct udev_monitor **pmon)
 {
 	if (*pmon) {
@@ -174,12 +181,64 @@ static int monitor_main_loop(struct udev_monitor *monitor)
 	return ret;
 }
 
+static const char autoconnect_rules[] = "/run/udev/rules.d/70-nvmf-autoconnect.rules";
+
+static int monitor_disable_udev_rules(void)
+{
+	CLEANUP(char, path) = strdup(autoconnect_rules);
+	char *s1, *s2;
+	int rc;
+
+	if (!path)
+		return -ENOMEM;
+
+	s2 = strrchr(path, '/');
+	for (s1 = s2 - 1; s1 > path && *s1 != '/'; s1--);
+
+	*s2 = *s1 = '\0';
+	rc = mkdir(path, 0755);
+	if (rc == 0 || errno == EEXIST) {
+		*s1 = '/';
+		rc = mkdir(path, 0755);
+		if (rc == 0 || errno == EEXIST) {
+			*s2 = '/';
+			rc = symlink("/dev/null", path);
+		}
+	}
+	if (rc) {
+		if (errno == EEXIST) {
+			char target[PATH_MAX];
+
+			if (readlink(path, target, sizeof(target)) != -1 &&
+			    !strcmp(target, "/dev/null")) {
+				log(LOG_INFO,
+				    "symlink %s -> /dev/null exists already\n",
+				    autoconnect_rules);
+				return 1;
+			}
+		}
+		log(LOG_ERR, "error creating %s: %m\n", autoconnect_rules);
+	} else
+		log(LOG_INFO, "created %s\n", autoconnect_rules);
+
+	return rc ? (errno ? -errno : -EIO) : 0;
+}
+
+static void monitor_enable_udev_rules(void)
+{
+	if (unlink(autoconnect_rules) == -1 && errno != ENOENT)
+		log(LOG_ERR, "error removing %s: %m\n", autoconnect_rules);
+	else
+		log(LOG_INFO, "removed %s\n", autoconnect_rules);
+}
+
 static int monitor_parse_opts(const char *desc, int argc, char **argv)
 {
 	bool quiet = false;
 	bool verbose = false;
 	bool debug = false;
-	int ret;
+	int ret = 0;
+
 	OPT_ARGS(opts) = {
 		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
 		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
@@ -198,7 +257,17 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 		log_level = LOG_INFO;
 	if (debug)
 		log_level = LOG_DEBUG;
-
+	if (mon_cfg.autoconnect) {
+		ret = monitor_disable_udev_rules();
+		if (ret < 0) {
+			mon_cfg.autoconnect = false;
+			log(LOG_WARNING, "autoconnect disabled\n");
+			ret = 0;
+		} else if (ret > 0) {
+			mon_cfg.skip_udev_on_exit = true;
+			ret = 0;
+		}
+	}
 	return ret;
 }
 
@@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc, char **argv)
 		udev_monitor_unref(monitor);
 	}
 	udev = udev_unref(udev);
+	if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
+		monitor_enable_udev_rules();
 out:
 	return nvme_status_to_errno(ret, true);
 }
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 14/35] monitor: implement handling of fc_udev_device

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The reason we fork off discovery tasks is two-fold: Firstly, we'd
otherwise be forced to make all discovery connections sequentially,
which would be slow, as connecting controllers can block on
the order of seconds even in successful cases.

Secondly, this allows us to use the some static variables like
cfg and tracked_ctrls in the discovery code path. Without forking,
we'd have to re-write much more code in fabrics.c. In general,
the alternative to forking would be creating threads, but the
that would probably require a large rewrite of our code base.

A single-threaded server that forks off the actual discovery makes
most sense at this point.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 107 insertions(+), 3 deletions(-)

diff --git a/monitor.c b/monitor.c
index 2a906db..6547265 100644
--- a/monitor.c
+++ b/monitor.c
@@ -30,7 +30,9 @@
 
 #include "common.h"
 #include "nvme-status.h"
+#include "nvme.h"
 #include "util/argconfig.h"
+#include "fabrics.h"
 #include "monitor.h"
 #define LOG_FUNCNAME 1
 #include "log.h"
@@ -117,11 +119,113 @@ static int monitor_init_signals(void)
 	return 0;
 }
 
+static int monitor_get_fc_uev_props(struct udev_device *ud,
+				    char *traddr, size_t tra_sz,
+				    char *host_traddr, size_t htra_sz)
+{
+	const char *sysname = udev_device_get_sysname(ud);
+	const char *tra = NULL, *host_tra = NULL;
+	bool fc_event_seen = false;
+	struct udev_list_entry *entry;
+
+	entry = udev_device_get_properties_list_entry(ud);
+	if (!entry) {
+		log(LOG_NOTICE, "%s: emtpy properties list\n", sysname);
+		return -ENOENT;
+	}
+
+	for (; entry; entry = udev_list_entry_get_next(entry)) {
+		const char *name = udev_list_entry_get_name(entry);
+
+		if (!strcmp(name, "FC_EVENT") &&
+		    !strcmp(udev_list_entry_get_value(entry), "nvmediscovery"))
+				fc_event_seen = true;
+		else if (!strcmp(name, "NVMEFC_HOST_TRADDR"))
+			host_tra = udev_list_entry_get_value(entry);
+		else if (!strcmp(name, "NVMEFC_TRADDR"))
+			tra = udev_list_entry_get_value(entry);
+	}
+	if (!fc_event_seen) {
+		log(LOG_DEBUG, "%s: FC_EVENT property missing or unsupported\n",
+		    sysname);
+		return -EINVAL;
+	}
+	if (!tra || !host_tra) {
+		log(LOG_WARNING, "%s: transport properties missing\n", sysname);
+		return -EINVAL;
+	}
+
+	if (!memccpy(traddr, tra, '\0', tra_sz) ||
+	    !memccpy(host_traddr, host_tra, '\0', htra_sz)) {
+		log(LOG_ERR, "traddr (%zu) or host_traddr (%zu) overflow\n",
+		    strlen(traddr), strlen(host_traddr));
+		return -ENAMETOOLONG;
+	}
+
+	return 0;
+}
+
+static int monitor_discovery(char *transport, char *traddr, char *trsvcid,
+			     char *host_traddr)
+{
+	char argstr[BUF_SIZE];
+	pid_t pid;
+	int rc;
+
+	pid = fork();
+	if (pid == -1) {
+		log(LOG_ERR, "failed to fork discovery task: %m");
+		return -errno;
+	} else if (pid > 0)
+		return 0;
+
+	log(LOG_NOTICE, "starting %s discovery for %s==>%s(%s)\n",
+	    transport, host_traddr, traddr, trsvcid ? trsvcid : "none");
+	cfg.nqn = NVME_DISC_SUBSYS_NAME;
+	cfg.transport = transport;
+	cfg.traddr = traddr;
+	cfg.trsvcid = trsvcid;
+	cfg.host_traddr = host_traddr;
+	/* Without the following, the kernel returns EINVAL */
+	cfg.tos = -1;
+
+	rc = build_options(argstr, sizeof(argstr), true);
+	log(LOG_DEBUG, "%s\n", argstr);
+	rc = do_discover(argstr, mon_cfg.autoconnect);
+
+	exit(-rc);
+	/* not reached */
+	return rc;
+}
+
+static void monitor_handle_fc_uev(struct udev_device *ud)
+{
+	const char *action = udev_device_get_action(ud);
+	const char *sysname = udev_device_get_sysname(ud);
+	char traddr[NVMF_TRADDR_SIZE], host_traddr[NVMF_TRADDR_SIZE];
+
+	if (strcmp(action, "change") || strcmp(sysname, "fc_udev_device"))
+		return;
+
+	if (monitor_get_fc_uev_props(ud, traddr, sizeof(traddr),
+				     host_traddr, sizeof(host_traddr)))
+		return;
+
+	monitor_discovery("fc", traddr, NULL, host_traddr);
+}
+
 static void monitor_handle_udevice(struct udev_device *ud)
 {
-	log(LOG_INFO, "uevent: %s %s\n",
-		udev_device_get_action(ud),
-		udev_device_get_sysname(ud));
+	const char *subsys  = udev_device_get_subsystem(ud);
+
+	if (log_level >= LOG_INFO) {
+		const char *action = udev_device_get_action(ud);
+		const char *syspath = udev_device_get_syspath(ud);
+
+		log(LOG_INFO, "%s %s\n", action, syspath);
+	}
+	if (!strcmp(subsys, "fc"))
+		monitor_handle_fc_uev(ud);
 }
 
 static void monitor_handle_uevents(struct udev_monitor *monitor)
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 15/35] monitor: implement handling of nvme AEN events

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This is very similar to fc_udev_device events, except that we
have different udev properties and must use trsvcid.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/monitor.c b/monitor.c
index 6547265..69c8333 100644
--- a/monitor.c
+++ b/monitor.c
@@ -214,6 +214,74 @@ static void monitor_handle_fc_uev(struct udev_device *ud)
 	monitor_discovery("fc", traddr, NULL, host_traddr);
 }
 
+static int monitor_get_nvme_uev_props(struct udev_device *ud,
+				      char *transport, size_t tr_sz,
+				      char *traddr, size_t tra_sz,
+				      char *trsvcid, size_t trs_sz,
+				      char *host_traddr, size_t htra_sz)
+{
+	const char *sysname = udev_device_get_sysname(ud);
+	bool aen_disc = false;
+	struct udev_list_entry *entry;
+
+	entry = udev_device_get_properties_list_entry(ud);
+	if (!entry) {
+		log(LOG_NOTICE, "%s: emtpy properties list\n", sysname);
+		return -ENOENT;
+	}
+
+	*transport = *traddr = *trsvcid = *host_traddr = '\0';
+	for (; entry; entry = udev_list_entry_get_next(entry)) {
+		const char *name = udev_list_entry_get_name(entry);
+
+		if (!strcmp(name, "NVME_AEN") &&
+		    !strcmp(udev_list_entry_get_value(entry), "0x70f002"))
+				aen_disc = true;
+		else if (!strcmp(name, "NVME_TRTYPE"))
+			memccpy(transport, udev_list_entry_get_value(entry),
+				'\0', tr_sz);
+		else if (!strcmp(name, "NVME_TRADDR"))
+			memccpy(traddr, udev_list_entry_get_value(entry),
+				'\0', htra_sz);
+		else if (!strcmp(name, "NVME_TRSVCID"))
+			memccpy(trsvcid, udev_list_entry_get_value(entry),
+				'\0', trs_sz);
+		else if (!strcmp(name, "NVME_HOST_TRADDR"))
+			memccpy(host_traddr, udev_list_entry_get_value(entry),
+				'\0', tra_sz);
+	}
+	if (!aen_disc) {
+		log(LOG_DEBUG, "%s: not a \"discovery log changed\" AEN, ignoring event\n",
+		    sysname);
+		return -EINVAL;
+	}
+
+	if (!*traddr || !*transport) {
+		log(LOG_WARNING, "%s: transport properties missing\n", sysname);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void monitor_handle_nvme_uev(struct udev_device *ud)
+{
+	char traddr[NVMF_TRADDR_SIZE], host_traddr[NVMF_TRADDR_SIZE];
+	char trsvcid[NVMF_TRSVCID_SIZE], transport[5];
+
+	if (strcmp(udev_device_get_action(ud), "change"))
+		return;
+
+	if (monitor_get_nvme_uev_props(ud, transport, sizeof(transport),
+				       traddr, sizeof(traddr),
+				       trsvcid, sizeof(trsvcid),
+				       host_traddr, sizeof(host_traddr)))
+		return;
+
+	monitor_discovery(transport, traddr,
+			  strcmp(trsvcid, "none") ? trsvcid : NULL, host_traddr);
+}
+
 static void monitor_handle_udevice(struct udev_device *ud)
 {
 	const char *subsys  = udev_device_get_subsystem(ud);
@@ -226,6 +294,8 @@ static void monitor_handle_udevice(struct udev_device *ud)
 	}
 	if (!strcmp(subsys, "fc"))
 		monitor_handle_fc_uev(ud);
+	else if (!strcmp(subsys, "nvme"))
+		monitor_handle_nvme_uev(ud);
 }
 
 static void monitor_handle_uevents(struct udev_monitor *monitor)
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 16/35] monitor: reset children's signal disposition

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index 69c8333..9375a14 100644
--- a/monitor.c
+++ b/monitor.c
@@ -100,6 +100,8 @@ static void monitor_int_handler(int sig)
 	must_exit = 1;
 }
 
+static sigset_t orig_sigmask;
+
 static int monitor_init_signals(void)
 {
 	sigset_t mask;
@@ -110,7 +112,7 @@ static int monitor_init_signals(void)
 	 * for events.
 	 */
 	sigfillset(&mask);
-	if (sigprocmask(SIG_BLOCK, &mask, NULL) == -1)
+	if (sigprocmask(SIG_BLOCK, &mask, &orig_sigmask) == -1)
 		return -errno;
 	if (sigaction(SIGTERM, &sa, NULL) == -1)
 		return -errno;
@@ -119,6 +121,26 @@ static int monitor_init_signals(void)
 	return 0;
 }
 
+static int child_reset_signals(void)
+{
+	int err = 0;
+	struct sigaction sa = { .sa_handler = SIG_DFL, };
+
+	if (sigaction(SIGTERM, &sa, NULL) == -1)
+		err = errno;
+	if (sigaction(SIGINT, &sa, NULL) == -1 && !err)
+		err = errno;
+	if (sigaction(SIGCHLD, &sa, NULL) == -1 && !err)
+		err = errno;
+
+	if (sigprocmask(SIG_SETMASK, &orig_sigmask, NULL) == -1 && !err)
+		err = errno;
+
+	if (err)
+		log(LOG_ERR, "error resetting signal handlers and mask\n");
+	return -err;
+}
+
 static int monitor_get_fc_uev_props(struct udev_device *ud,
 				    char *traddr, size_t tra_sz,
 				    char *host_traddr, size_t htra_sz)
@@ -176,8 +198,12 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,
 	if (pid == -1) {
 		log(LOG_ERR, "failed to fork discovery task: %m");
 		return -errno;
-	} else if (pid > 0)
+	} else if (pid > 0) {
+		log(LOG_DEBUG, "started discovery task %ld\n", (long)pid);
 		return 0;
+	}
+
+	child_reset_signals();
 
 	log(LOG_NOTICE, "starting %s discovery for %s==>%s(%s)\n",
 	    transport, host_traddr, traddr, trsvcid ? trsvcid : "none");
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 17/35] monitor: handle SIGCHLD for terminated child processes

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 48 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9375a14..8db40e6 100644
--- a/monitor.c
+++ b/monitor.c
@@ -26,6 +26,8 @@
 #include <limits.h>
 #include <syslog.h>
 #include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/wait.h>
 #include <sys/epoll.h>
 
 #include "common.h"
@@ -100,6 +102,13 @@ static void monitor_int_handler(int sig)
 	must_exit = 1;
 }
 
+static sig_atomic_t got_sigchld;
+
+static void monitor_chld_handler(int sig)
+{
+	got_sigchld = 1;
+}
+
 static sigset_t orig_sigmask;
 
 static int monitor_init_signals(void)
@@ -118,6 +127,9 @@ static int monitor_init_signals(void)
 		return -errno;
 	if (sigaction(SIGINT, &sa, NULL) == -1)
 		return -errno;
+	sa.sa_handler = monitor_chld_handler;
+	if (sigaction(SIGCHLD, &sa, NULL) == -1)
+		return -errno;
 	return 0;
 }
 
@@ -336,6 +348,34 @@ static void monitor_handle_uevents(struct udev_monitor *monitor)
 	}
 }
 
+static void handle_sigchld(void)
+{
+	while (true) {
+		int wstatus;
+		pid_t pid;
+
+		pid = waitpid(-1, &wstatus, WNOHANG);
+		switch(pid) {
+		case -1:
+			if (errno != ECHILD)
+				log(LOG_ERR, "error in waitpid: %m\n");
+			return;
+		case 0:
+			return;
+		default:
+			break;
+		}
+		if (!WIFEXITED(wstatus))
+			log(LOG_WARNING, "child %ld didn't exit normally\n",
+			    (long)pid);
+		else if (WEXITSTATUS(wstatus) != 0)
+			log(LOG_NOTICE, "child %ld exited with status \"%s\"\n",
+			    (long)pid, strerror(WEXITSTATUS(wstatus)));
+		else
+			log(LOG_DEBUG, "child %ld exited normally\n", (long)pid);
+	};
+}
+
 #define MAX_EVENTS 1
 static int monitor_main_loop(struct udev_monitor *monitor)
 {
@@ -357,13 +397,19 @@ static int monitor_main_loop(struct udev_monitor *monitor)
 	sigfillset(&ep_mask);
 	sigdelset(&ep_mask, SIGTERM);
 	sigdelset(&ep_mask, SIGINT);
+	sigdelset(&ep_mask, SIGCHLD);
 	while (1) {
 		int rc, i;
 
 		rc = epoll_pwait(ep_fd, events, MAX_EVENTS, -1, &ep_mask);
 		if (rc == -1 && errno == EINTR) {
-			log(LOG_NOTICE, "monitor: exit signal received\n");
-			return 0;
+			if (must_exit) {
+				log(LOG_NOTICE, "monitor: exit signal received\n");
+				return 0;
+			} else if (got_sigchld) {
+				got_sigchld = 0;
+				handle_sigchld();
+			}
 		} else if (rc == -1) {
 			log(LOG_ERR, "monitor: epoll_wait: %m\n");
 			return -errno;
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 18/35] monitor: add "--persistent/-p" flag

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Without this flag, AENs on created controllers won't be received.
While --persistent is probably the behavior most users will want,
it has to be explicitly activated, to be consistent with the
connect-all command.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/monitor.c b/monitor.c
index 8db40e6..419c5f5 100644
--- a/monitor.c
+++ b/monitor.c
@@ -487,6 +487,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 
 	OPT_ARGS(opts) = {
 		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
+		OPT_FLAG("persistent",     'p', &cfg.persistent,      "persistent discovery connections"),
 		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
 		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
 		OPT_FLAG("debug",          'D', &debug,               "log level: debug"),
@@ -514,6 +515,10 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 			ret = 0;
 		}
 	}
+
+	if (cfg.persistent && !cfg.keep_alive_tmo)
+		cfg.keep_alive_tmo = NVMF_DEF_DISC_TMO;
+
 	return ret;
 }
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 19/35] fabrics: use "const char *" in struct config

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This is easily done, and allows passing in some const char* pointers
that would otherwise need to be strdup()d first.

constifying cfg->device requires changes in the ctrl_instance()
function, because basename() can't be used.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 fabrics.c | 23 ++++++++++++++---------
 fabrics.h | 17 +++++++++--------
 2 files changed, 23 insertions(+), 17 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index c1a4bb5..e3d2a3a 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -252,17 +252,22 @@ empty_field:
 	return strdup("\0");
 }
 
-static int ctrl_instance(char *device)
+int ctrl_instance(const char *device)
 {
 	char d[64];
+	const char *p;
 	int ret, instance;
 
-	device = basename(device);
-	ret = sscanf(device, "nvme%d", &instance);
+	p = strrchr(device, '/');
+	if (p == NULL)
+		p = device;
+	else
+		p++;
+	ret = sscanf(p, "nvme%d", &instance);
 	if (ret <= 0)
 		return -EINVAL;
 	if (snprintf(d, sizeof(d), "nvme%d", instance) <= 0 ||
-	    strcmp(device, d))
+	    strcmp(p, d))
 		return -EINVAL;
 	return instance;
 }
@@ -273,7 +278,7 @@ static int ctrl_instance(char *device)
  * given.
  * Return true/false based on whether it matches
  */
-static bool ctrl_matches_connectargs(char *name, struct connect_args *args)
+static bool ctrl_matches_connectargs(const char *name, struct connect_args *args)
 {
 	struct connect_args cargs;
 	bool found = false;
@@ -815,7 +820,7 @@ add_int_argument(char **argstr, int *max_len, char *arg_str, int arg,
 }
 
 static int
-add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
+add_argument(char **argstr, int *max_len, char *arg_str, const char *arg)
 {
 	int len;
 
@@ -1541,7 +1546,7 @@ static int scan_sys_nvme_filter(const struct dirent *d)
 /*
  * Returns 1 if disconnect occurred, 0 otherwise.
  */
-static int disconnect_subsys(char *nqn, char *ctrl)
+static int disconnect_subsys(const char *nqn, char *ctrl)
 {
 	char *sysfs_nqn_path = NULL, *sysfs_del_path = NULL;
 	char subsysnqn[NVMF_NQN_SIZE] = {};
@@ -1579,7 +1584,7 @@ static int disconnect_subsys(char *nqn, char *ctrl)
 /*
  * Returns the number of controllers successfully disconnected.
  */
-static int disconnect_by_nqn(char *nqn)
+static int disconnect_by_nqn(const char *nqn)
 {
 	struct dirent **devices = NULL;
 	int i, n, ret = 0;
@@ -1601,7 +1606,7 @@ static int disconnect_by_nqn(char *nqn)
 	return ret;
 }
 
-static int disconnect_by_device(char *device)
+static int disconnect_by_device(const char *device)
 {
 	int instance;
 
diff --git a/fabrics.h b/fabrics.h
index ce965a3..1dfbd67 100644
--- a/fabrics.h
+++ b/fabrics.h
@@ -13,13 +13,13 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);
 /* Symbols used by monitor.c */
 
 struct config {
-	char *nqn;
-	char *transport;
-	char *traddr;
-	char *trsvcid;
-	char *host_traddr;
-	char *hostnqn;
-	char *hostid;
+	const char *nqn;
+	const char *transport;
+	const char *traddr;
+	const char *trsvcid;
+	const char *host_traddr;
+	const char *hostnqn;
+	const char *hostid;
 	int  nr_io_queues;
 	int  nr_write_queues;
 	int  nr_poll_queues;
@@ -29,7 +29,7 @@ struct config {
 	int  ctrl_loss_tmo;
 	int  tos;
 	char *raw;
-	char *device;
+	const char *device;
 	int  duplicate_connect;
 	int  disable_sqflow;
 	int  hdr_digest;
@@ -43,5 +43,6 @@ extern struct config cfg;
 
 int build_options(char *argstr, int max_len, bool discover);
 int do_discover(char *argstr, bool connect);
+int ctrl_instance(const char *device);
 
 #endif
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 20/35] fabrics: export arg_str(), parse_conn_arg(), and remove_ctrl()

[mwilck]

From: Martin Wilck <mwilck@suse.com>

These functions are used by the monitor functionality that will
be added in follow-up commits.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 fabrics.c | 6 +++---
 fabrics.h | 4 ++++
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index e3d2a3a..e2629d5 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -102,7 +102,7 @@ static const match_table_t opt_tokens = {
 	{ OPT_ERR,		NULL		},
 };
 
-static const char *arg_str(const char * const *strings,
+const char *arg_str(const char * const *strings,
 		size_t array_size, size_t idx)
 {
 	if (idx < array_size && strings[idx])
@@ -208,7 +208,7 @@ int do_discover(char *argstr, bool connect);
  * If field found, return string containing field value. If field
  * not found, return an empty string.
  */
-static char *parse_conn_arg(char *conargs, const char delim, const char *field)
+char *parse_conn_arg(char *conargs, const char delim, const char *field)
 {
 	char *s, *e;
 	size_t cnt;
@@ -476,7 +476,7 @@ out:
 	return ret;
 }
 
-static int remove_ctrl(int instance)
+int remove_ctrl(int instance)
 {
 	char *sysfs_path;
 	int ret;
diff --git a/fabrics.h b/fabrics.h
index 1dfbd67..ddc7423 100644
--- a/fabrics.h
+++ b/fabrics.h
@@ -12,6 +12,8 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);
 
 /* Symbols used by monitor.c */
 
+const char *arg_str(const char * const *strings, size_t array_size, size_t idx);
+
 struct config {
 	const char *nqn;
 	const char *transport;
@@ -44,5 +46,7 @@ extern struct config cfg;
 int build_options(char *argstr, int max_len, bool discover);
 int do_discover(char *argstr, bool connect);
 int ctrl_instance(const char *device);
+char *parse_conn_arg(char *conargs, const char delim, const char *field);
+int remove_ctrl(int instance);
 
 #endif
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 21/35] nvme-cli: add "list.h"

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Adding "list.h" from multipath-tools, which got it from the kernel
long ago.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 list.h | 365 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 365 insertions(+)
 create mode 100644 list.h

diff --git a/list.h b/list.h
new file mode 100644
index 0000000..ced021f
--- /dev/null
+++ b/list.h
@@ -0,0 +1,365 @@
+/*
+ * Copied from the Linux kernel source tree, version 2.6.0-test1.
+ *
+ * Licensed under the GPL v2 as per the whole kernel source tree.
+ *
+ */
+
+#ifndef _LIST_H
+#define _LIST_H
+
+#include <stddef.h>
+
+/**
+ * container_of - cast a member of a structure out to the containing structure
+ *
+ * @ptr:	the pointer to the member.
+ * @type:	the type of the container struct this is embedded in.
+ * @member:	the name of the member within the struct.
+ *
+ */
+#define container_of_const(ptr, type, member) ({		\
+	const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
+	(const type *)( (const char *)__mptr - offsetof(type,member) );})
+
+#define container_of(ptr, type, member) ({		\
+	typeof( ((type *)0)->member ) *__mptr = (ptr);	\
+	(type *)( (char *)__mptr - offsetof(type,member) );})
+
+/*
+ * These are non-NULL pointers that will result in page faults
+ * under normal circumstances, used to verify that nobody uses
+ * non-initialized list entries.
+ */
+#define LIST_POISON1  ((void *) 0x00100100)
+#define LIST_POISON2  ((void *) 0x00200200)
+
+/*
+ * Simple doubly linked list implementation.
+ *
+ * Some of the internal functions ("__xxx") are useful when
+ * manipulating whole lists rather than single entries, as
+ * sometimes we already know the next/prev entries and we can
+ * generate better code by using them directly rather than
+ * using the generic single-entry routines.
+ */
+
+struct list_head {
+	struct list_head *next, *prev;
+};
+
+#define LIST_HEAD_INIT(name) { &(name), &(name) }
+
+#define LIST_HEAD(name) \
+	struct list_head name = LIST_HEAD_INIT(name)
+
+#define INIT_LIST_HEAD(ptr) do { \
+	(ptr)->next = (ptr); (ptr)->prev = (ptr); \
+} while (0)
+
+/*
+ * Insert a new entry between two known consecutive entries.
+ *
+ * This is only for internal list manipulation where we know
+ * the prev/next entries already!
+ */
+static inline void __list_add(struct list_head *new,
+			      struct list_head *prev,
+			      struct list_head *next)
+{
+	next->prev = new;
+	new->next = next;
+	new->prev = prev;
+	prev->next = new;
+}
+
+/**
+ * list_add - add a new entry
+ * @new: new entry to be added
+ * @head: list head to add it after
+ *
+ * Insert a new entry after the specified head.
+ * This is good for implementing stacks.
+ */
+static inline void list_add(struct list_head *new, struct list_head *head)
+{
+	__list_add(new, head, head->next);
+}
+
+/**
+ * list_add_tail - add a new entry
+ * @new: new entry to be added
+ * @head: list head to add it before
+ *
+ * Insert a new entry before the specified head.
+ * This is useful for implementing queues.
+ */
+static inline void list_add_tail(struct list_head *new, struct list_head *head)
+{
+	__list_add(new, head->prev, head);
+}
+
+/*
+ * Delete a list entry by making the prev/next entries
+ * point to each other.
+ *
+ * This is only for internal list manipulation where we know
+ * the prev/next entries already!
+ */
+static inline void __list_del(struct list_head * prev, struct list_head * next)
+{
+	next->prev = prev;
+	prev->next = next;
+}
+
+/**
+ * list_del - deletes entry from list.
+ * @entry: the element to delete from the list.
+ * Note: list_empty on entry does not return true after this, the entry is
+ * in an undefined state.
+ */
+static inline void list_del(struct list_head *entry)
+{
+	__list_del(entry->prev, entry->next);
+	entry->next = LIST_POISON1;
+	entry->prev = LIST_POISON2;
+}
+
+/**
+ * list_del_init - deletes entry from list and reinitialize it.
+ * @entry: the element to delete from the list.
+ */
+static inline void list_del_init(struct list_head *entry)
+{
+	__list_del(entry->prev, entry->next);
+	INIT_LIST_HEAD(entry);
+}
+
+/**
+ * list_move - delete from one list and add as another's head
+ * @list: the entry to move
+ * @head: the head that will precede our entry
+ */
+static inline void list_move(struct list_head *list, struct list_head *head)
+{
+	__list_del(list->prev, list->next);
+	list_add(list, head);
+}
+
+/**
+ * list_move_tail - delete from one list and add as another's tail
+ * @list: the entry to move
+ * @head: the head that will follow our entry
+ */
+static inline void list_move_tail(struct list_head *list,
+				  struct list_head *head)
+{
+	__list_del(list->prev, list->next);
+	list_add_tail(list, head);
+}
+
+/**
+ * list_empty - tests whether a list is empty
+ * @head: the list to test.
+ */
+static inline int list_empty(struct list_head *head)
+{
+	return head->next == head;
+}
+
+static inline void __list_splice(const struct list_head *list,
+				 struct list_head *prev,
+				 struct list_head *next)
+{
+	struct list_head *first = list->next;
+	struct list_head *last = list->prev;
+
+	first->prev = prev;
+	prev->next = first;
+
+	last->next = next;
+	next->prev = last;
+}
+
+/**
+ * list_splice - join two lists
+ * @list: the new list to add.
+ * @head: the place to add it in the first list.
+ */
+static inline void list_splice(struct list_head *list, struct list_head *head)
+{
+	if (!list_empty(list))
+		__list_splice(list, head, head->next);
+}
+
+/**
+ * list_splice_tail - join two lists, each list being a queue
+ * @list: the new list to add.
+ * @head: the place to add it in the first list.
+ */
+static inline void list_splice_tail(struct list_head *list,
+				    struct list_head *head)
+{
+	if (!list_empty(list))
+		__list_splice(list, head->prev, head);
+}
+
+/**
+ * list_splice_init - join two lists and reinitialise the emptied list.
+ * @list: the new list to add.
+ * @head: the place to add it in the first list.
+ *
+ * The list at @list is reinitialised
+ */
+static inline void list_splice_init(struct list_head *list,
+				    struct list_head *head)
+{
+	if (!list_empty(list)) {
+		__list_splice(list, head, head->next);
+		INIT_LIST_HEAD(list);
+	}
+}
+
+/**
+ * list_splice_tail_init - join two lists and reinitialise the emptied list
+ * @list: the new list to add.
+ * @head: the place to add it in the first list.
+ *
+ * Each of the lists is a queue.
+ * The list at @list is reinitialised
+ */
+static inline void list_splice_tail_init(struct list_head *list,
+					 struct list_head *head)
+{
+	if (!list_empty(list)) {
+		__list_splice(list, head->prev, head);
+		INIT_LIST_HEAD(list);
+	}
+}
+
+/**
+ * list_entry - get the struct for this entry
+ * @ptr:	the &struct list_head pointer.
+ * @type:	the type of the struct this is embedded in.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_entry(ptr, type, member) \
+	container_of(ptr, type, member)
+
+/**
+ * list_for_each	-	iterate over a list
+ * @pos:	the &struct list_head to use as a loop counter.
+ * @head:	the head for your list.
+ */
+#define list_for_each(pos, head) \
+	for (pos = (head)->next; pos != (head); \
+		pos = pos->next)
+
+/**
+ * __list_for_each	-	iterate over a list
+ * @pos:	the &struct list_head to use as a loop counter.
+ * @head:	the head for your list.
+ *
+ * This variant differs from list_for_each() in that it's the
+ * simplest possible list iteration code.
+ * Use this for code that knows the list to be very short (empty
+ * or 1 entry) most of the time.
+ */
+#define __list_for_each(pos, head) \
+	for (pos = (head)->next; pos != (head); pos = pos->next)
+
+/**
+ * list_for_each_prev	-	iterate over a list backwards
+ * @pos:	the &struct list_head to use as a loop counter.
+ * @head:	the head for your list.
+ */
+#define list_for_each_prev(pos, head) \
+	for (pos = (head)->prev; pos != (head); pos = pos->prev)
+
+/**
+ * list_for_each_safe	-	iterate over a list safe against removal of list entry
+ * @pos:	the &struct list_head to use as a loop counter.
+ * @n:		another &struct list_head to use as temporary storage
+ * @head:	the head for your list.
+ */
+#define list_for_each_safe(pos, n, head) \
+	for (pos = (head)->next, n = pos->next; pos != (head); \
+		pos = n, n = pos->next)
+
+/**
+ * list_for_each_entry	-	iterate over list of given type
+ * @pos:	the type * to use as a loop counter.
+ * @head:	the head for your list.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_each_entry(pos, head, member)				\
+	for (pos = list_entry((head)->next, typeof(*pos), member);	\
+	     &pos->member != (head);					\
+	     pos = list_entry(pos->member.next, typeof(*pos), member))
+
+/**
+ * list_for_each_entry_reverse - iterate backwards over list of given type.
+ * @pos:	the type * to use as a loop counter.
+ * @head:	the head for your list.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_each_entry_reverse(pos, head, member)			\
+	for (pos = list_entry((head)->prev, typeof(*pos), member);	\
+	     &pos->member != (head);					\
+	     pos = list_entry(pos->member.prev, typeof(*pos), member))
+
+/**
+ * list_for_each_entry_safe - iterate over list of given type safe against removal of list entry
+ * @pos:	the type * to use as a loop counter.
+ * @n:		another type * to use as temporary storage
+ * @head:	the head for your list.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_each_entry_safe(pos, n, head, member)			\
+	for (pos = list_entry((head)->next, typeof(*pos), member),	\
+		n = list_entry(pos->member.next, typeof(*pos), member);	\
+	     &pos->member != (head);					\
+	     pos = n, n = list_entry(n->member.next, typeof(*n), member))
+
+/**
+ * list_for_each_entry_reverse_safe - iterate backwards over list of given type safe against removal of list entry
+ * @pos:	the type * to use as a loop counter.
+ * @n:		another type * to use as temporary storage
+ * @head:	the head for your list.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_each_entry_reverse_safe(pos, n, head, member)          \
+	for (pos = list_entry((head)->prev, typeof(*pos), member),      \
+		 n = list_entry(pos->member.prev, typeof(*pos), member);\
+	     &pos->member != (head);                                    \
+	     pos = n, n = list_entry(n->member.prev, typeof(*n), member))
+
+/**
+ * list_for_some_entry_safe - iterate list from the given begin node to the given end node safe against removal of list entry
+ * @pos:	the type * to use as a loop counter.
+ * @n:		another type * to use as temporary storage
+ * @from:	the begin node of the iteration.
+ * @to:		the end node of the iteration.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_some_entry_safe(pos, n, from, to, member)              \
+	for (pos = list_entry((from)->next, typeof(*pos), member),      \
+	     n = list_entry(pos->member.next, typeof(*pos), member);    \
+	     &pos->member != (to);                                      \
+	     pos = n, n = list_entry(n->member.next, typeof(*n), member))
+
+/**
+ * list_for_some_entry_reverse_safe - iterate backwards list from the given begin node to the given end node safe against removal of list entry
+ * @pos:	the type * to use as a loop counter.
+ * @n:		another type * to use as temporary storage
+ * @from:	the begin node of the iteration.
+ * @to:		the end node of the iteration.
+ * @member:	the name of the list_struct within the struct.
+ */
+#define list_for_some_entry_reverse_safe(pos, n, from, to, member)      \
+	for (pos = list_entry((from)->prev, typeof(*pos), member),      \
+	     n = list_entry(pos->member.prev, typeof(*pos), member);    \
+	     &pos->member != (to);                                      \
+	     pos = n, n = list_entry(n->member.prev, typeof(*n), member))
+
+#endif /* _LIST_H */
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 22/35] conn-db: add simple connection registry

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The monitor works best if it maintains a discovery controller connection
to every transport address that provides a discovery subsystem.

While controllers are easily tracked in sysfs, addresses ("connections"),
i.e. (transport, traddr, trsvid, host_traddr) tuples, are not. Create
a simple registry that tracks the state of "connections" and their
associated discovery controllers.

A detailed description of the API is provided in the header file conn-db.h.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 Makefile  |   2 +-
 conn-db.c | 340 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 conn-db.h | 140 ++++++++++++++++++++++
 3 files changed, 481 insertions(+), 1 deletion(-)
 create mode 100644 conn-db.c
 create mode 100644 conn-db.h

diff --git a/Makefile b/Makefile
index 21c9a23..6ac5030 100644
--- a/Makefile
+++ b/Makefile
@@ -69,7 +69,7 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \
 	nvme-status.o nvme-filters.o nvme-topology.o
 
 ifeq ($(HAVE_LIBUDEV),0)
-	OBJS += monitor.o
+	OBJS += monitor.o conn-db.o
 endif
 
 UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o
diff --git a/conn-db.c b/conn-db.c
new file mode 100644
index 0000000..99d88da
--- /dev/null
+++ b/conn-db.c
@@ -0,0 +1,340 @@
+/*
+ * Copyright (C) 2021 SUSE LLC
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ *
+ * This file implements a simple registry for NVMe connections, i.e.
+ * (transport type, host_traddr, traddr, trsvcid) tuples.
+ */
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <syslog.h>
+#include <time.h>
+
+#include "common.h"
+#include "list.h"
+#include "nvme.h"
+#include "fabrics.h"
+#include "conn-db.h"
+
+#define LOG_FUNCNAME 1
+#include "log.h"
+
+struct conn_int {
+	struct nvme_connection c;
+	struct list_head lst;
+};
+
+#define conn2internal(co) container_of(co, struct conn_int, c)
+
+static LIST_HEAD(connections);
+
+CLEANUP_FUNC(char)
+
+static const char * const _status_str[] = {
+	[CS_NEW] = "new",
+	[CS_DISC_RUNNING] = "discovery-running",
+	[CS_ONLINE] = "online",
+	[CS_FAILED] = "failed",
+};
+
+const char *conn_status_str(int status)
+{
+	return arg_str(_status_str, ARRAY_SIZE(_status_str), status);
+}
+
+#define _log_conn(lvl, msg, transport, traddr, trsvcid, host_traddr)	\
+	do {								\
+		const char *__trs = trsvcid;				\
+									\
+		log(lvl, "%s <%s>: %s ==> %s(%s)\n",			\
+		    msg, transport, host_traddr, traddr,		\
+		    __trs && *__trs ? __trs : "none");			\
+	} while (0)
+
+#define log_conn(lvl, msg, conn)					\
+	_log_conn(lvl, msg, (conn)->c.transport,			\
+		  (conn)->c.traddr, (conn)->c.trsvcid,			\
+		  (conn)->c.host_traddr)
+
+static void conn_free(struct conn_int *ci)
+{
+	if (!ci)
+		return;
+	if (ci->c.traddr)
+		free(ci->c.traddr);
+	if (ci->c.trsvcid)
+		free(ci->c.trsvcid);
+	if (ci->c.host_traddr)
+		free(ci->c.host_traddr);
+	free(ci);
+}
+
+static void conn_free_p(struct conn_int **ci)
+{
+	if (*ci) {
+		conn_free(*ci);
+		*ci = NULL;
+	}
+}
+
+static int conn_del(struct conn_int *ci)
+{
+	if (!ci)
+		return -ENOENT;
+	if (list_empty(&ci->lst))
+		return -EINVAL;
+	log_conn(LOG_INFO, "forgetting connection", ci);
+	list_del(&ci->lst);
+	conn_free(ci);
+	return 0;
+}
+
+bool conndb_matches(const char *transport, const char *traddr,
+		    const char *trsvcid, const char *host_traddr,
+		    const struct nvme_connection *co)
+{
+	if (!co)
+		return false;
+	if (!transport || strcmp(transport, co->transport))
+		return false;
+	if (!traddr || strncmp(traddr, co->traddr, NVMF_TRADDR_SIZE))
+		return false;
+	if ((!trsvcid && co->trsvcid) ||
+	    (trsvcid && *trsvcid && (!co->trsvcid ||
+			 strncmp(trsvcid, co->trsvcid, NVMF_TRSVCID_SIZE))))
+		return false;
+	if (!host_traddr || (strncmp(host_traddr, co->host_traddr,
+				     NVMF_TRADDR_SIZE)))
+		return false;
+	return true;
+}
+
+static struct conn_int *conn_find(const char *transport, const char *traddr,
+				  const char *trsvcid, const char *host_traddr)
+{
+	struct conn_int *ci;
+
+	if (!transport || !traddr || !host_traddr)
+		return NULL;
+	list_for_each_entry(ci, &connections, lst) {
+		if (conndb_matches(transport, traddr, trsvcid, host_traddr, &ci->c))
+			return ci;
+	}
+	return NULL;
+}
+
+static bool is_supported_transport(const char *transport)
+{
+
+	return !strcmp(transport, "fc") || !strcmp(transport, "rdma") ||
+	       !strcmp(transport, "tcp") || !strcmp(transport, "loop");
+}
+
+static int _conn_add(const char *transport, const char *traddr,
+		     const char *trsvcid, const char *host_traddr,
+		     struct conn_int **new_ci)
+{
+	struct conn_int *ci __attribute__((cleanup(conn_free_p))) = NULL;
+
+	if (!transport || !is_supported_transport(transport) || !traddr)
+		return -EINVAL;
+
+	if (!(ci = calloc(1, sizeof(*ci))) ||
+	    !(ci->c.traddr = strndup(traddr, NVMF_TRADDR_SIZE)) ||
+	    !(ci->c.host_traddr = strndup(host_traddr, NVMF_TRADDR_SIZE)) ||
+	    (trsvcid && *trsvcid &&
+	     !(ci->c.trsvcid = strndup(trsvcid, NVMF_TRSVCID_SIZE))))
+		return -ENOMEM;
+	memccpy(ci->c.transport, transport, '\0', sizeof(ci->c.transport));
+	ci->c.status = CS_NEW;
+	ci->c.discovery_instance = -1;
+	list_add(&ci->lst, &connections);
+	*new_ci = ci;
+	ci = NULL;
+	return 0;
+}
+
+static int conn_add(const char *transport, const char *traddr,
+		    const char *trsvcid, const char *host_traddr,
+		    struct conn_int **new_ci)
+{
+	struct conn_int *ci = conn_find(transport, traddr, trsvcid, host_traddr);
+	int rc;
+
+	if (ci) {
+		*new_ci = ci;
+		return -EEXIST;
+	}
+	rc = _conn_add(transport, traddr, trsvcid, host_traddr, new_ci);
+	if (!rc)
+		log_conn(LOG_DEBUG, "added connection", *new_ci);
+	else
+		_log_conn(LOG_ERR, "failed to add", transport, traddr,
+			  trsvcid, host_traddr);
+	return rc;
+}
+
+int conndb_add(const char *transport, const char *traddr,
+	       const char *trsvcid, const char *host_traddr,
+	       struct nvme_connection **new_conn)
+{
+	struct conn_int *ci = NULL;
+	int rc = conn_add(transport, traddr, trsvcid, host_traddr, &ci);
+
+	if (rc != 0 && rc != -EEXIST)
+		return rc;
+	if (new_conn)
+		*new_conn = &ci->c;
+	return rc;
+}
+
+struct nvme_connection *conndb_find(const char *transport, const char *traddr,
+				    const char *trsvcid, const char *host_traddr)
+{
+	struct conn_int *ci;
+
+	ci = conn_find(transport, traddr, trsvcid, host_traddr);
+	if (ci)
+		return &ci->c;
+	else
+		return NULL;
+}
+
+struct nvme_connection *conndb_find_by_pid(pid_t pid)
+{
+	struct conn_int *ci;
+
+	list_for_each_entry(ci, &connections, lst) {
+		if (ci->c.status == CS_DISC_RUNNING &&
+		    ci->c.discovery_task == pid)
+			return &ci->c;
+	}
+	return NULL;
+}
+
+struct nvme_connection *conndb_find_by_ctrl(const char *devname)
+{
+	struct conn_int *ci;
+	int instance;
+
+	instance = ctrl_instance(devname);
+	if (!instance)
+		return NULL;
+
+	list_for_each_entry(ci, &connections, lst) {
+		if (ci->c.discovery_instance == instance)
+			return &ci->c;
+	}
+	return NULL;
+}
+
+int conndb_delete(struct nvme_connection *co)
+{
+	if (!co)
+		return -ENOENT;
+	return conn_del(conn2internal(co));
+}
+
+void conndb_free(void)
+{
+	struct conn_int *ci, *next;
+
+	list_for_each_entry_safe(ci, next, &connections, lst)
+		conn_del(ci);
+}
+
+int conndb_init_from_sysfs(void)
+{
+	struct dirent **devices;
+	int i, n, ret = 0;
+	char syspath[PATH_MAX];
+
+	n = scandir(SYS_NVME, &devices, scan_ctrls_filter, alphasort);
+	if (n <= 0)
+		return n;
+
+	for (i = 0; i < n; i++) {
+		int len, rc;
+		struct conn_int *ci;
+		CLEANUP(char, transport) = NULL;
+		CLEANUP(char, address) = NULL;
+		CLEANUP(char, traddr) = NULL;
+		CLEANUP(char, trsvcid) = NULL;
+		CLEANUP(char, host_traddr) = NULL;
+		CLEANUP(char, subsysnqn) = NULL;
+
+		len = snprintf(syspath, sizeof(syspath), SYS_NVME "/%s",
+			       devices[i]->d_name);
+		if (len < 0 || len >= sizeof(syspath))
+			continue;
+
+		transport = nvme_get_ctrl_attr(syspath, "transport");
+		address = nvme_get_ctrl_attr(syspath, "address");
+		if (!transport || !address)
+			continue;
+		traddr = parse_conn_arg(address, ' ', "traddr");
+		trsvcid = parse_conn_arg(address, ' ', "trsvcid");
+		host_traddr = parse_conn_arg(address, ' ', "host_traddr");
+
+		rc = conn_add(transport, traddr, trsvcid, host_traddr, &ci);
+		if (rc != 0 && rc != -EEXIST)
+			continue;
+
+		if (rc == 0)
+			ret ++;
+		subsysnqn = nvme_get_ctrl_attr(syspath, "subsysnqn");
+		if (subsysnqn && !strcmp(subsysnqn, NVME_DISC_SUBSYS_NAME)) {
+			int instance = ctrl_instance(devices[i]->d_name);
+
+			if (instance >= 0) {
+				ci->c.discovery_instance = instance;
+				log(LOG_DEBUG, "found discovery controller %s\n",
+				    devices[i]->d_name);
+			}
+		}
+		free(devices[i]);
+	}
+	free(devices);
+	return ret;
+}
+
+int conndb_for_each(int (*callback)(struct nvme_connection *co, void *arg),
+		    void *arg)
+{
+	struct conn_int *ci, *next;
+	int ret = 0;
+
+	list_for_each_entry_safe(ci, next, &connections, lst) {
+		int rc = callback(&ci->c, arg);
+
+		if (rc & ~(CD_CB_ERR|CD_CB_DEL|CD_CB_BREAK)) {
+			log(LOG_ERR,
+			    "invalid return value 0x%x from callback\n", rc);
+			ret = -EINVAL;
+			continue;
+		}
+		if (rc & CD_CB_ERR) {
+			log(LOG_WARNING, "callback returned error\n");
+			if (!ret)
+				ret = errno ? -errno : -EIO;
+		}
+		if (rc & CD_CB_DEL)
+			conn_del(ci);
+		if (rc & CD_CB_BREAK)
+			break;
+	}
+	return ret;
+}
diff --git a/conn-db.h b/conn-db.h
new file mode 100644
index 0000000..c599c15
--- /dev/null
+++ b/conn-db.h
@@ -0,0 +1,140 @@
+#ifndef _CONN_DB_H
+#define _CONN_DB_H
+
+struct nvme_connection {
+	char transport[5];
+	char *traddr;
+	char *trsvcid;
+	char *host_traddr;
+
+	int status;
+	int discovery_pending:1;
+	int did_discovery:1;
+	int successful_discovery:1;
+	union {
+		pid_t discovery_task;
+		int discovery_result;
+	};
+	int discovery_instance;
+};
+
+/* connection status */
+enum {
+	CS_NEW = 0,
+	CS_DISC_RUNNING,
+	CS_ONLINE,
+	CS_FAILED,
+	__CS_LAST,
+};
+
+/**
+ * conn_status_str() - return string representation of connection status
+ */
+const char *conn_status_str(int status);
+
+/**
+ * conndb_add() - add a connection with given parameters
+ *
+ * @new_conn: if non-NULL and the function succeeds, will receive a pointer
+ *            to the either existing or newly created connection object.
+ *
+ * Looks up the given connection parameters in the db and adds a new connection
+ * unless found. All input parameters except trsvcid must be non-NULL.
+ *
+ * Return: 0 if controller was added, -EEXIST if controller existed in the db
+ *         (this is considered success), or other negative error code in
+ *         the error case.
+ *
+ */
+int conndb_add(const char *transport, const char *traddr,
+	       const char *trsvcid, const char *host_traddr,
+	       struct nvme_connection **new_conn);
+
+/**
+ * conndb_find() - lookup a connection with given parameters
+ *
+ * Return: NULL if not found, valid connection object otherwise.
+ */
+struct nvme_connection *conndb_find(const char *transport, const char *traddr,
+				    const char *trsvcid, const char *host_traddr);
+
+
+/**
+ * conndb_find_by_pid() - lookup connection by discovery task pid
+ *
+ * Return: valid connetion object if successful, NULL otherwise.
+ */
+struct nvme_connection *conndb_find_by_pid(pid_t pid);
+
+
+/**
+ * conndb_find_by_pid() - lookup connection from controller instance
+ *
+ * Return: valid connetion object if a connection was found that has
+ * the given device as discovery controller. NULL otherwise.
+ */
+struct nvme_connection *conndb_find_by_ctrl(const char *devname);
+
+enum {
+	CD_CB_OK    = 0,
+	CD_CB_ERR   = (1 << 0),
+	CD_CB_DEL   = (1 << 1),
+	CD_CB_BREAK = (1 << 2),
+};
+
+/**
+ *  conndb_for_each() - run a callback for each connection
+ *
+ * @callback: function to be called
+ * @arg:      user argument passed to callback
+ *
+ * The callback must return a bitmask created from the CD_CB_* enum
+ * values above. CD_CB_ERR signals an error condition in the callback.
+ * CD_CB_DEL causes the connection to be deleted after the callback
+ * returns. CD_CB_BREAK stops the iteration. Returning a value that
+ * is not an OR-ed from these values is an error.
+ *
+ * Return: 0 if all callbacks completed successfully.
+ *         A negative error code if some callback failed.
+ */
+int conndb_for_each(int (*callback)(struct nvme_connection *co, void *arg),
+		    void *arg);
+
+/**
+ * conndb_matches - check if connection matches given parameters
+ *
+ * The arguments @transport and @traddr must be non-null and non-empty.
+ * @trscvid and @host_traddr may be NULL, in which case they match
+ * connections that don't have these attributes set, either.
+ *
+ * Return: true iff the given connection matches the given attributes.
+ */
+bool conndb_matches(const char *transport, const char *traddr,
+		    const char *trsvcid, const char *host_traddr,
+		    const struct nvme_connection *co);
+
+/**
+ * conndb_delete() - remove a given nvme connection object
+ *
+ * Removes the object from the data base and frees it.
+ *
+ * Return: 0 if successful, negative error code otherwise
+ */
+int conndb_delete(struct nvme_connection *co);
+
+/**
+ * conndb-free() - free internal data structures
+ */
+void conndb_free(void);
+
+/**
+ * conndb_init_from_sysfs() - check existing NVMe connections
+ *
+ * Populates the connection db from existing contoller devices in sysfs.
+ *
+ * Return: (positive or zero) number of found connections on success.
+ *         Negative error code on failure.
+ */
+int conndb_init_from_sysfs(void);
+
+#endif
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 23/35] monitor: handle restart of pending discoveries

[mwilck]

From: Martin Wilck <mwilck@suse.com>

If a new discovery event is received while a discovery task is
running, no new task should be started in parallel to the previous
one. Rather, we should note that another event has happened,
and restart the discovery after the current task has finished.

There is no need to remember more than one pending discovery.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 49 insertions(+), 9 deletions(-)

diff --git a/monitor.c b/monitor.c
index 419c5f5..b44ff7a 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,6 +36,7 @@
 #include "util/argconfig.h"
 #include "fabrics.h"
 #include "monitor.h"
+#include "conn-db.h"
 #define LOG_FUNCNAME 1
 #include "log.h"
 
@@ -199,12 +200,22 @@ static int monitor_get_fc_uev_props(struct udev_device *ud,
 	return 0;
 }
 
-static int monitor_discovery(char *transport, char *traddr, char *trsvcid,
-			     char *host_traddr)
+static int monitor_discovery(const char *transport, const char *traddr,
+			     const char *trsvcid, const char *host_traddr)
 {
 	char argstr[BUF_SIZE];
 	pid_t pid;
-	int rc;
+	int rc, db_rc;
+	struct nvme_connection *co = NULL;
+
+	db_rc = conndb_add(transport, traddr, trsvcid, host_traddr, &co);
+	if (db_rc != 0 && db_rc != -EEXIST)
+		return db_rc;
+
+	if (co->status == CS_DISC_RUNNING) {
+		co->discovery_pending = 1;
+		return -EAGAIN;
+	}
 
 	pid = fork();
 	if (pid == -1) {
@@ -212,17 +223,26 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,
 		return -errno;
 	} else if (pid > 0) {
 		log(LOG_DEBUG, "started discovery task %ld\n", (long)pid);
+
+		co->discovery_pending = 0;
+		co->status = CS_DISC_RUNNING;
+		co->discovery_task = pid;
+
 		return 0;
 	}
 
 	child_reset_signals();
 
-	log(LOG_NOTICE, "starting %s discovery for %s==>%s(%s)\n",
-	    transport, host_traddr, traddr, trsvcid ? trsvcid : "none");
+	log(LOG_NOTICE, "starting discovery for <%s>: %s ==> %s(%s) in state %s\n",
+	    transport, host_traddr, traddr,
+	    trsvcid && *trsvcid ? trsvcid : "none",
+	    conn_status_str(co->status));
+
+
 	cfg.nqn = NVME_DISC_SUBSYS_NAME;
 	cfg.transport = transport;
 	cfg.traddr = traddr;
-	cfg.trsvcid = trsvcid;
+	cfg.trsvcid = trsvcid && *trsvcid ? trsvcid : NULL;
 	cfg.host_traddr = host_traddr;
 	/* Without the following, the kernel returns EINVAL */
 	cfg.tos = -1;
@@ -351,6 +371,7 @@ static void monitor_handle_uevents(struct udev_monitor *monitor)
 static void handle_sigchld(void)
 {
 	while (true) {
+	struct nvme_connection *co;
 		int wstatus;
 		pid_t pid;
 
@@ -365,14 +386,33 @@ static void handle_sigchld(void)
 		default:
 			break;
 		}
-		if (!WIFEXITED(wstatus))
+		co = conndb_find_by_pid(pid);
+		if (!co) {
+			log(LOG_ERR, "no connection found for discovery task %ld\n",
+			    (long)pid);
+			continue;
+		}
+		if (!WIFEXITED(wstatus)) {
 			log(LOG_WARNING, "child %ld didn't exit normally\n",
 			    (long)pid);
-		else if (WEXITSTATUS(wstatus) != 0)
+			co->status = CS_FAILED;
+		} else if (WEXITSTATUS(wstatus) != 0) {
 			log(LOG_NOTICE, "child %ld exited with status \"%s\"\n",
 			    (long)pid, strerror(WEXITSTATUS(wstatus)));
-		else
+			co->status = CS_FAILED;
+			co->did_discovery = 1;
+			co->discovery_result = WEXITSTATUS(wstatus);
+		} else {
 			log(LOG_DEBUG, "child %ld exited normally\n", (long)pid);
+			co->status = CS_ONLINE;
+			co->successful_discovery = co->did_discovery = 1;
+			co->discovery_result = 0;
+		}
+		if (co->discovery_pending) {
+			log(LOG_NOTICE, "new discovery pending - restarting\n");
+			monitor_discovery(co->transport, co->traddr,
+					  co->trsvcid, co->host_traddr);
+		}
 	};
 }
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 24/35] monitor: monitor_discovery(): try to reuse existing controllers

[mwilck]

From: Martin Wilck <mwilck@suse.com>

We need to pass cfg.device to do_discovery() if we want to re-use the
discovery controller device. Remember the instance number when
handling an AEN, and try to reuse this instance if the discovery
needs to be repeated.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 36 +++++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)

diff --git a/monitor.c b/monitor.c
index b44ff7a..ce772f3 100644
--- a/monitor.c
+++ b/monitor.c
@@ -201,12 +201,14 @@ static int monitor_get_fc_uev_props(struct udev_device *ud,
 }
 
 static int monitor_discovery(const char *transport, const char *traddr,
-			     const char *trsvcid, const char *host_traddr)
+			     const char *trsvcid, const char *host_traddr,
+			     const char *devname)
 {
 	char argstr[BUF_SIZE];
 	pid_t pid;
 	int rc, db_rc;
 	struct nvme_connection *co = NULL;
+	char *device = NULL;
 
 	db_rc = conndb_add(transport, traddr, trsvcid, host_traddr, &co);
 	if (db_rc != 0 && db_rc != -EEXIST)
@@ -227,7 +229,15 @@ static int monitor_discovery(const char *transport, const char *traddr,
 		co->discovery_pending = 0;
 		co->status = CS_DISC_RUNNING;
 		co->discovery_task = pid;
+		if (devname) {
+			int instance = ctrl_instance(devname);
 
+			if (instance < 0) {
+				log(LOG_ERR, "unexpected devname: %s\n",
+				    devname);
+			} else
+				co->discovery_instance = instance;
+		}
 		return 0;
 	}
 
@@ -238,12 +248,26 @@ static int monitor_discovery(const char *transport, const char *traddr,
 	    trsvcid && *trsvcid ? trsvcid : "none",
 	    conn_status_str(co->status));
 
+	/*
+	 * Try to re-use existing controller. do_discovery() will check
+	 * if it matches the connection parameters.
+	 */
+	if (!devname && co->discovery_instance >= 0) {
+		if (asprintf(&device, "nvme%d", co->discovery_instance) == -1)
+			device = NULL;
+		else
+			devname = device;
+	}
+
+	if (devname)
+		log(LOG_INFO, "using discovery controller %s\n", devname);
 
 	cfg.nqn = NVME_DISC_SUBSYS_NAME;
 	cfg.transport = transport;
 	cfg.traddr = traddr;
 	cfg.trsvcid = trsvcid && *trsvcid ? trsvcid : NULL;
-	cfg.host_traddr = host_traddr;
+	cfg.host_traddr = host_traddr && *host_traddr ? host_traddr : NULL;
+	cfg.device = devname;
 	/* Without the following, the kernel returns EINVAL */
 	cfg.tos = -1;
 
@@ -251,6 +275,7 @@ static int monitor_discovery(const char *transport, const char *traddr,
 	log(LOG_DEBUG, "%s\n", argstr);
 	rc = do_discover(argstr, mon_cfg.autoconnect);
 
+	free(device);
 	exit(-rc);
 	/* not reached */
 	return rc;
@@ -269,7 +294,7 @@ static void monitor_handle_fc_uev(struct udev_device *ud)
 				     host_traddr, sizeof(host_traddr)))
 		return;
 
-	monitor_discovery("fc", traddr, NULL, host_traddr);
+	monitor_discovery("fc", traddr, NULL, host_traddr, NULL);
 }
 
 static int monitor_get_nvme_uev_props(struct udev_device *ud,
@@ -337,7 +362,8 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)
 		return;
 
 	monitor_discovery(transport, traddr,
-			  strcmp(trsvcid, "none") ? trsvcid : NULL, host_traddr);
+			  strcmp(trsvcid, "none") ? trsvcid : NULL, host_traddr,
+			  udev_device_get_sysname(ud));
 }
 
 static void monitor_handle_udevice(struct udev_device *ud)
@@ -411,7 +437,7 @@ static void handle_sigchld(void)
 		if (co->discovery_pending) {
 			log(LOG_NOTICE, "new discovery pending - restarting\n");
 			monitor_discovery(co->transport, co->traddr,
-					  co->trsvcid, co->host_traddr);
+					  co->trsvcid, co->host_traddr, NULL);
 		}
 	};
 }
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 25/35] monitor: read existing connections on startup

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Add discovery_ctrl_existed flag to the connection, to determine
whether a given (discovery) controller should be shut down on exit.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 conn-db.c | 1 +
 conn-db.h | 1 +
 monitor.c | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/conn-db.c b/conn-db.c
index 99d88da..07e4dde 100644
--- a/conn-db.c
+++ b/conn-db.c
@@ -301,6 +301,7 @@ int conndb_init_from_sysfs(void)
 
 			if (instance >= 0) {
 				ci->c.discovery_instance = instance;
+				ci->c.discovery_ctrl_existed = 1;
 				log(LOG_DEBUG, "found discovery controller %s\n",
 				    devices[i]->d_name);
 			}
diff --git a/conn-db.h b/conn-db.h
index c599c15..11c3502 100644
--- a/conn-db.h
+++ b/conn-db.h
@@ -11,6 +11,7 @@ struct nvme_connection {
 	int discovery_pending:1;
 	int did_discovery:1;
 	int successful_discovery:1;
+	int discovery_ctrl_existed:1;
 	union {
 		pid_t discovery_task;
 		int discovery_result;
diff --git a/monitor.c b/monitor.c
index ce772f3..5bd900b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -603,9 +603,11 @@ int aen_monitor(const char *desc, int argc, char **argv)
 	}
 	ret = create_udev_monitor(&monitor);
 	if (ret == 0) {
+		conndb_init_from_sysfs();
 		ret = monitor_main_loop(monitor);
 		udev_monitor_unref(monitor);
 	}
+	conndb_free();
 	udev = udev_unref(udev);
 	if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
 		monitor_enable_udev_rules();
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 26/35] monitor: implement starting discovery controllers on startup

[mwilck]

From: Martin Wilck <mwilck@suse.com>

If the -U/--startup option is used, nvme monitor looks for exsiting
nvme controllers on startup, and runs a discovery on the associated
transport address (connection); i.e. tries to connect to a discovery
subsystem on the same address, retrieve the discovery log pages,
and (if --autoconnect is given) connect all listed controllers.

It depends on the --persistent option whether the discovery controllers
created in this process will be persistent after startup.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/monitor.c b/monitor.c
index 5bd900b..e5d1982 100644
--- a/monitor.c
+++ b/monitor.c
@@ -43,6 +43,7 @@
 static struct monitor_config {
 	bool autoconnect;
 	bool skip_udev_on_exit;
+	bool start_ctrls;
 } mon_cfg;
 
 static struct udev *udev;
@@ -544,6 +545,20 @@ static void monitor_enable_udev_rules(void)
 		log(LOG_INFO, "removed %s\n", autoconnect_rules);
 }
 
+static int monitor_setup_discovery_ctrl(struct nvme_connection *co,
+					void *arg __attribute__((unused)))
+{
+	int rc;
+
+	if (co->discovery_instance >= 0)
+		return CD_CB_OK;
+
+	rc = monitor_discovery(co->transport, co->traddr, co->trsvcid,
+			       co->host_traddr, NULL);
+
+	return rc ? CD_CB_ERR : CD_CB_OK;
+}
+
 static int monitor_parse_opts(const char *desc, int argc, char **argv)
 {
 	bool quiet = false;
@@ -553,6 +568,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 
 	OPT_ARGS(opts) = {
 		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
+		OPT_FLAG("startup",        'U', &mon_cfg.start_ctrls, "start discovery controllers on startup"),
 		OPT_FLAG("persistent",     'p', &cfg.persistent,      "persistent discovery connections"),
 		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
 		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
@@ -604,6 +620,8 @@ int aen_monitor(const char *desc, int argc, char **argv)
 	ret = create_udev_monitor(&monitor);
 	if (ret == 0) {
 		conndb_init_from_sysfs();
+		if (mon_cfg.start_ctrls)
+			conndb_for_each(monitor_setup_discovery_ctrl, NULL);
 		ret = monitor_main_loop(monitor);
 		udev_monitor_unref(monitor);
 	}
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 27/35] monitor: implement cleanup of created discovery controllers

[mwilck]

From: Martin Wilck <mwilck@suse.com>

If -A or -U was in use, try to clean up discovery controllers we set
up during runtime. Discovery connections that were already up when
the monitor started will not be teared down.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/monitor.c b/monitor.c
index e5d1982..e3fee55 100644
--- a/monitor.c
+++ b/monitor.c
@@ -559,6 +559,39 @@ static int monitor_setup_discovery_ctrl(struct nvme_connection *co,
 	return rc ? CD_CB_ERR : CD_CB_OK;
 }
 
+static int monitor_remove_discovery_ctrl(struct nvme_connection *co,
+					void *arg __attribute__((unused)))
+{
+	char syspath[PATH_MAX];
+	int len;
+	CLEANUP(char, subsysnqn) = NULL;
+
+	if (co->discovery_instance == -1 || co->discovery_ctrl_existed)
+		return CD_CB_OK;
+
+	len = snprintf(syspath, sizeof(syspath), SYS_NVME "/nvme%d",
+		       co->discovery_instance);
+	if (len < 0 || len >= sizeof(syspath))
+		return CD_CB_ERR;
+
+	subsysnqn = nvme_get_ctrl_attr(syspath, "subsysnqn");
+	if (subsysnqn && !strcmp(subsysnqn, NVME_DISC_SUBSYS_NAME)) {
+		if (remove_ctrl(co->discovery_instance)) {
+			log(LOG_ERR,
+			    "failed to remove discovery controller /dev/nvme%d: %m\n",
+			    co->discovery_instance);
+			return CD_CB_ERR;
+		} else
+			log(LOG_INFO,
+			    "removed discovery controller /dev/nvme%d\n",
+			    co->discovery_instance);
+	} else
+		log(LOG_WARNING,
+		    "unexpected NQN %s on /dev/nvme%d, not removing controller\n",
+		    subsysnqn, co->discovery_instance);
+	return CD_CB_OK;
+}
+
 static int monitor_parse_opts(const char *desc, int argc, char **argv)
 {
 	bool quiet = false;
@@ -625,6 +658,8 @@ int aen_monitor(const char *desc, int argc, char **argv)
 		ret = monitor_main_loop(monitor);
 		udev_monitor_unref(monitor);
 	}
+	if (mon_cfg.autoconnect || mon_cfg.start_ctrls)
+		conndb_for_each(monitor_remove_discovery_ctrl, NULL);
 	conndb_free();
 	udev = udev_unref(udev);
 	if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 28/35] monitor: basic handling of add/remove uevents for nvme controllers

[mwilck]

From: Martin Wilck <mwilck@suse.com>

On add, we'd like to detect the creation of a discovery controller for a given
connection. More often than not, that doesn't work, because the controller
is still in "connecting" state when the uevent arrives, and reading subsysnqn
yields "(efault)".

This would need to be fixed by waiting for the connection to complete,
but doing this properly is currently out of scope. It's not really a big
problem, because without --persistent, the discovery controller will
be removed anyway, and with --persistent, we'll notice the discovery
controller as soon as it actually receives an AEN and generates an uevent.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/monitor.c b/monitor.c
index e3fee55..e2c2568 100644
--- a/monitor.c
+++ b/monitor.c
@@ -155,6 +155,36 @@ static int child_reset_signals(void)
 	return -err;
 }
 
+static void monitor_handle_nvme_add(struct udev_device *ud)
+{
+	const char *syspath = udev_device_get_syspath(ud);
+	CLEANUP(char, subsysnqn) = NULL;
+	CLEANUP(char, state) = NULL;
+
+	if (!syspath)
+		return;
+	subsysnqn = nvme_get_ctrl_attr(syspath, "subsysnqn");
+	state = nvme_get_ctrl_attr(syspath, "state");
+	log(LOG_DEBUG, "add %s => %s [%s]\n", syspath, subsysnqn, state);
+}
+
+static void monitor_handle_nvme_remove(struct udev_device *ud)
+{
+	const char *sysname = udev_device_get_sysname(ud);
+	struct nvme_connection *co;
+
+	if (!sysname)
+		return;
+
+	co = conndb_find_by_ctrl(sysname);
+	if (co) {
+		log(LOG_DEBUG, "%s: connection discovery controller removed\n",
+		    sysname);
+		co->discovery_instance = -1;
+	}
+	return;
+}
+
 static int monitor_get_fc_uev_props(struct udev_device *ud,
 				    char *traddr, size_t tra_sz,
 				    char *host_traddr, size_t htra_sz)
@@ -353,6 +383,14 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)
 	char traddr[NVMF_TRADDR_SIZE], host_traddr[NVMF_TRADDR_SIZE];
 	char trsvcid[NVMF_TRSVCID_SIZE], transport[5];
 
+	if (!strcmp(udev_device_get_action(ud), "remove")) {
+		monitor_handle_nvme_remove(ud);
+		return;
+	}
+	if (!strcmp(udev_device_get_action(ud), "add")) {
+		monitor_handle_nvme_add(ud);
+		return;
+	}
 	if (strcmp(udev_device_get_action(ud), "change"))
 		return;
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 29/35] monitor: kill running discovery tasks on exit

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Make sure discovery tasks terminate when the main process terminates.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/monitor.c b/monitor.c
index e2c2568..37c9ace 100644
--- a/monitor.c
+++ b/monitor.c
@@ -630,6 +630,34 @@ static int monitor_remove_discovery_ctrl(struct nvme_connection *co,
 	return CD_CB_OK;
 }
 
+static int monitor_kill_discovery_task(struct nvme_connection *co,
+				       void *arg __attribute__((unused)))
+{
+	int wstatus;
+	pid_t pid, wpid = -1;
+
+	if (co->status != CS_DISC_RUNNING)
+		return CD_CB_OK;
+
+	pid = co->discovery_task;
+	co->status = CS_FAILED;
+	if (kill(co->discovery_task, SIGTERM) == -1) {
+		log(LOG_ERR, "failed to send SIGTERM to pid %ld: %m\n",
+		    (long)pid);
+		wpid = waitpid(pid, &wstatus, WNOHANG);
+	} else {
+		log(LOG_DEBUG, "sent SIGTERM to pid %ld, waiting\n", (long)pid);
+		wpid = waitpid(pid, &wstatus, 0);
+	}
+	if (wpid != pid) {
+		log(LOG_ERR, "failed to wait for %ld: %m\n", (long)pid);
+		return CD_CB_ERR;
+	} else {
+		log(LOG_DEBUG, "child %ld terminated\n", (long)pid);
+		return CD_CB_OK;
+	}
+}
+
 static int monitor_parse_opts(const char *desc, int argc, char **argv)
 {
 	bool quiet = false;
@@ -696,9 +724,12 @@ int aen_monitor(const char *desc, int argc, char **argv)
 		ret = monitor_main_loop(monitor);
 		udev_monitor_unref(monitor);
 	}
+
+	conndb_for_each(monitor_kill_discovery_task, NULL);
 	if (mon_cfg.autoconnect || mon_cfg.start_ctrls)
 		conndb_for_each(monitor_remove_discovery_ctrl, NULL);
 	conndb_free();
+
 	udev = udev_unref(udev);
 	if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
 		monitor_enable_udev_rules();
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 30/35] monitor: add connection property options from connect-all

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This code is simply copied from fabrics_discover(). That's possible
because we simply access the fabric "cfg" variable directly.
These parameters are assumed to be constant over the runtime
of the monitor.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 monitor.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/monitor.c b/monitor.c
index 37c9ace..ecf6706 100644
--- a/monitor.c
+++ b/monitor.c
@@ -669,6 +669,19 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
 		OPT_FLAG("startup",        'U', &mon_cfg.start_ctrls, "start discovery controllers on startup"),
 		OPT_FLAG("persistent",     'p', &cfg.persistent,      "persistent discovery connections"),
+		OPT_LIST("hostnqn",        'q', &cfg.hostnqn,         "user-defined hostnqn (if default not used)"),
+		OPT_LIST("hostid",         'I', &cfg.hostid,          "user-defined hostid (if default not used)"),
+		OPT_INT("keep-alive-tmo",  'k', &cfg.keep_alive_tmo,  "keep alive timeout period in seconds"),
+		OPT_INT("reconnect-delay", 'c', &cfg.reconnect_delay, "reconnect timeout period in seconds"),
+		OPT_INT("ctrl-loss-tmo",   'l', &cfg.ctrl_loss_tmo,   "controller loss timeout period in seconds"),
+		OPT_INT("tos",             'T', &cfg.tos,             "type of service"),
+		OPT_FLAG("hdr_digest",     'g', &cfg.hdr_digest,      "enable transport protocol header digest (TCP transport)"),
+		OPT_FLAG("data_digest",    'G', &cfg.data_digest,     "enable transport protocol data digest (TCP transport)"),
+		OPT_INT("nr-io-queues",    'i', &cfg.nr_io_queues,    "number of io queues to use (default is core count)"),
+		OPT_INT("nr-write-queues", 'W', &cfg.nr_write_queues, "number of write queues to use (default 0)"),
+		OPT_INT("nr-poll-queues",  'P', &cfg.nr_poll_queues,  "number of poll queues to use (default 0)"),
+		OPT_INT("queue-size",      'Q', &cfg.queue_size,      "number of io queue elements to use (default 128)"),
+		OPT_FLAG("matching",       'm', &cfg.matching_only,   "connect only records matching the traddr"),
 		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
 		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
 		OPT_FLAG("debug",          'D', &debug,               "log level: debug"),
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 31/35] completions: add completions for nvme monitor

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 completions/bash-nvme-completion.sh | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/completions/bash-nvme-completion.sh b/completions/bash-nvme-completion.sh
index d669e9b..9bf0da8 100644
--- a/completions/bash-nvme-completion.sh
+++ b/completions/bash-nvme-completion.sh
@@ -11,7 +11,7 @@ _cmds="list id-ctrl id-ns list-ns id-iocs create-ns delete-ns \
 	resv-report dsm flush compare read write write-zeroes \
 	write-uncor copy reset subsystem-reset show-regs discover \
 	connect-all connect disconnect version help \
-	intel lnvm memblaze list-subsys"
+	intel lnvm memblaze list-subsys monitor"
 
 nvme_list_opts () {
         local opts=""
@@ -231,6 +231,10 @@ nvme_list_opts () {
 		"disconnect")
 		opts+=" --nqn -n --device -d"
 			;;
+		"monitor")
+		opts+=" --autoconnect -A --persistent -p --startup -U \
+			--hostnqn= -q --verbose -v --clockstamps -C"
+			;;
 		"version")
 		opts+=""
 			;;
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 32/35] nvmf-autoconnect: add unit file for nvme-monitor.service

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 nvmf-autoconnect/systemd/nvme-monitor.service | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service

diff --git a/nvmf-autoconnect/systemd/nvme-monitor.service b/nvmf-autoconnect/systemd/nvme-monitor.service
new file mode 100644
index 0000000..d85b133
--- /dev/null
+++ b/nvmf-autoconnect/systemd/nvme-monitor.service
@@ -0,0 +1,17 @@
+[Unit]
+Description=NVMe Event Monitor for Automatical Subsystem Connection
+Documentation=man:nvme-monitor(1)
+DefaultDependencies=false
+Requires=systemd-udevd-kernel.socket
+RequiresMountsFor=/sys
+Conflicts=shutdown.target
+After=systemd-udevd-kernel.socket
+Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-connections.service
+
+[Service]
+Type=simple
+ExecStart=/usr/sbin/nvme monitor -A -p
+KillMode=mixed
+
+[Install]
+WantedBy=sysinit.target
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 33/35] nvme-connect-all(1): fix documentation for --quiet/-S

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 Documentation/nvme-connect-all.1    |  8 ++++----
 Documentation/nvme-connect-all.html | 10 +++++-----
 Documentation/nvme-connect-all.txt  |  4 ++--
 3 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/Documentation/nvme-connect-all.1 b/Documentation/nvme-connect-all.1
index 91d6896..a53a017 100644
--- a/Documentation/nvme-connect-all.1
+++ b/Documentation/nvme-connect-all.1
@@ -2,12 +2,12 @@
 .\"     Title: nvme-connect-all
 .\"    Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author]
 .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/>
-.\"      Date: 10/20/2020
+.\"      Date: 01/20/2021
 .\"    Manual: NVMe Manual
 .\"    Source: NVMe
 .\"  Language: English
 .\"
-.TH "NVME\-CONNECT\-ALL" "1" "10/20/2020" "NVMe" "NVMe Manual"
+.TH "NVME\-CONNECT\-ALL" "1" "01/20/2021" "NVMe" "NVMe Manual"
 .\" -----------------------------------------------------------------
 .\" * Define some portability stuff
 .\" -----------------------------------------------------------------
@@ -51,7 +51,7 @@ nvme-connect-all \- Discover and Connect to Fabrics controllers\&.
                 [\-\-queue\-size=<#>         | \-Q <#>]
                 [\-\-matching               | \-m]
                 [\-\-persistent             | \-p]
-                [\-\-quiet                  | \-q]
+                [\-\-quiet                  | \-S]
 .fi
 .SH "DESCRIPTION"
 .sp
@@ -193,7 +193,7 @@ If a traddr was specified on the command line or in the configuration file, only
 Don\(cqt remove the discovery controller after retrieving the discovery log page\&.
 .RE
 .PP
-\-q, \-\-quiet
+\-S, \-\-quiet
 .RS 4
 Suppress error messages\&.
 .RE
diff --git a/Documentation/nvme-connect-all.html b/Documentation/nvme-connect-all.html
index beae013..f779c7b 100644
--- a/Documentation/nvme-connect-all.html
+++ b/Documentation/nvme-connect-all.html
@@ -4,7 +4,7 @@
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
 <head>
 <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
-<meta name="generator" content="AsciiDoc 8.6.10" />
+<meta name="generator" content="AsciiDoc" />
 <title>nvme-connect-all(1)</title>
 <style type="text/css">
 /* Shared CSS for AsciiDoc xhtml11 and html5 backends */
@@ -436,7 +436,7 @@ thead, p.table.header {
 p.table {
   margin-top: 0;
 }
-/* Because the table frame attribute is overriden by CSS in most browsers. */
+/* Because the table frame attribute is overridden by CSS in most browsers. */
 div.tableblock > table[frame="void"] {
   border-style: none;
 }
@@ -768,7 +768,7 @@ nvme-connect-all(1) Manual Page
                 [--queue-size=&lt;#&gt;         | -Q &lt;#&gt;]
                 [--matching               | -m]
                 [--persistent             | -p]
-                [--quiet                  | -q]</pre>
+                [--quiet                  | -S]</pre>
 <div class="attribution">
 </div></div>
 </div>
@@ -1048,7 +1048,7 @@ cellspacing="0" cellpadding="4">
 </p>
 </dd>
 <dt class="hdlist1">
--q
+-S
 </dt>
 <dt class="hdlist1">
 --quiet
@@ -1115,7 +1115,7 @@ nvme-connect(1)</p></div>
 <div id="footer">
 <div id="footer-text">
 Last updated
- 2020-10-20 16:47:21 PDT
+ 2021-01-20 23:40:57 CET
 </div>
 </div>
 </body>
diff --git a/Documentation/nvme-connect-all.txt b/Documentation/nvme-connect-all.txt
index 128f336..820dd6c 100644
--- a/Documentation/nvme-connect-all.txt
+++ b/Documentation/nvme-connect-all.txt
@@ -27,7 +27,7 @@ SYNOPSIS
 		[--queue-size=<#>         | -Q <#>]
 		[--matching               | -m]
 		[--persistent             | -p]
-		[--quiet                  | -q]
+		[--quiet                  | -S]
 
 DESCRIPTION
 -----------
@@ -154,7 +154,7 @@ OPTIONS
 	Don't remove the discovery controller after retrieving the discovery
 	log page.
 
--q::
+-S::
 --quiet::
 	Suppress error messages.
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 34/35] nvme-monitor(1): add man page for nvme-monitor

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 Documentation/cmds-main.txt     |    4 +
 Documentation/nvme-monitor.1    |  204 ++++++
 Documentation/nvme-monitor.html | 1049 +++++++++++++++++++++++++++++++
 Documentation/nvme-monitor.txt  |  169 +++++
 4 files changed, 1426 insertions(+)
 create mode 100644 Documentation/nvme-monitor.1
 create mode 100644 Documentation/nvme-monitor.html
 create mode 100644 Documentation/nvme-monitor.txt

diff --git a/Documentation/cmds-main.txt b/Documentation/cmds-main.txt
index 46df03d..d058a54 100644
--- a/Documentation/cmds-main.txt
+++ b/Documentation/cmds-main.txt
@@ -168,3 +168,7 @@ linknvme:nvme-disconnect-all[1]::
 
 linknvme:nvme-get-property[1]::
 	Reads and shows NVMe-over-Fabrics controller property
+
+linknvme:nvme-monitor[1]::
+	Monitor Discovery events and Discover and Connect automatically
+
diff --git a/Documentation/nvme-monitor.1 b/Documentation/nvme-monitor.1
new file mode 100644
index 0000000..1578e4c
--- /dev/null
+++ b/Documentation/nvme-monitor.1
@@ -0,0 +1,204 @@
+'\" t
+.\"     Title: nvme-monitor
+.\"    Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author]
+.\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/>
+.\"      Date: 01/20/2021
+.\"    Manual: NVMe Manual
+.\"    Source: NVMe
+.\"  Language: English
+.\"
+.TH "NVME\-MONITOR" "1" "01/20/2021" "NVMe" "NVMe Manual"
+.\" -----------------------------------------------------------------
+.\" * Define some portability stuff
+.\" -----------------------------------------------------------------
+.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.\" http://bugs.debian.org/507673
+.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
+.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.ie \n(.g .ds Aq \(aq
+.el       .ds Aq '
+.\" -----------------------------------------------------------------
+.\" * set default formatting
+.\" -----------------------------------------------------------------
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.\" -----------------------------------------------------------------
+.\" * MAIN CONTENT STARTS HERE *
+.\" -----------------------------------------------------------------
+.SH "NAME"
+nvme-monitor \- Monitor Discovery events and Discover and Connect automatically
+.SH "SYNOPSIS"
+.sp
+.nf
+\fInvme discover\fR
+                [\-\-autoconnect            | \-A
+                [\-\-startup                | \-U
+                [\-\-persistent             | \-p
+                [\-\-hostnqn=<hostnqn>      | \-q <hostnqn>]
+                [\-\-hostid=<hostid>        | \-I <hostid>]
+                [\-\-keep\-alive\-tmo=<#>     | \-k <#>]
+                [\-\-reconnect\-delay=<#>    | \-c <#>]
+                [\-\-ctrl\-loss\-tmo=<#>      | \-l <#>]
+                [\-\-hdr\-digest             | \-g]
+                [\-\-data\-digest            | \-G]
+                [\-\-nr\-io\-queues=<#>       | \-i <#>]
+                [\-\-nr\-write\-queues=<#>    | \-W <#>]
+                [\-\-nr\-poll\-queues=<#>     | \-P <#>]
+                [\-\-queue\-size=<#>         | \-Q <#>]
+                [\-\-matching               | \-m]
+                [\-\-persistent             | \-p]
+                [\-\-silent                 | \-S]
+                [\-\-verbose                | \-v]
+                [\-\-debug                  | \-D]
+                [\-\-clockstamps            | \-C]
+.fi
+.SH "DESCRIPTION"
+.sp
+Listen to Discovery events (Asynchronous Event Notifications, AENs) on NVMe\-over\-Fabrics (NVMeoF) Discovery Controllers and for other events related to NVMeoF Discovery, and optionally connect to newly discovered controllers\&.
+.sp
+If no parameters are given, then \fInvme monitor\fR listens to Discovery\-related udev events (uevents)\&. If an event is received, it connects to the Discovery Controller and performs the equivalent of an \fInvme discover\fR on the associated transport address\&. If the \fI\-\-autoconnect\fR option is given, it performs the equivalent of an \fInvme connect\-all\fR instead\&. When run through a systemd service in \fI\-\-autoconnect\fR mode, the monitor can be used as an alternative to the udev\-rule based auto\-activation of NVMeoF connections\&.
+.sp
+Currently, the following event types are supported:
+.sp
+.RS 4
+.ie n \{\
+\h'-04'\(bu\h'+03'\c
+.\}
+.el \{\
+.sp -1
+.IP \(bu 2.3
+.\}
+"Discovery Log Page Change" Asynchronous Event Notifications (AENs) delivered via persistent connections to NVMeoF discovery controllers connected to the discovery service (nqn\&.2014\-08\&.org\&.nvmexpress\&.discovery)\&. Note that without the
+\fI\-\-persistent\fR
+option,
+\fInvme monitor\fR
+will not create persistent Discovery controllers itself, but it will receive and act upon AENs sent from persistent Discovery Controllers which were created manually or by other programs\&.
+.RE
+.sp
+.RS 4
+.ie n \{\
+\h'-04'\(bu\h'+03'\c
+.\}
+.el \{\
+.sp -1
+.IP \(bu 2.3
+.\}
+FC\-NVMe auto\-connect uevents sent when the FibreChannel transport discovers N_Ports offering NVMe services\&.
+.RE
+.sp
+See the documentation for the nvme\-connect\-all(1) and nvme\-discover(1) commands for further background\&.
+.SH "OPTIONS"
+.PP
+\-A, \-\-autoconnect
+.RS 4
+If this option is given,
+\fInvme monitor\fR
+will attempt to connect to newly discovered controllers\&. In order to avoid race conditions with
+\fInvme connect\-all\fR
+commands spawned by udev rules via systemd\(cqs
+nvmf\-connect@\&.service, an empty rule is created under
+/run/udev/rules\&.d/70\-nvmf\-autoconnect\&.rules, disabling the udev rules that start this service\&. The file will be deleted when
+\fInvme monitor\fR
+terminates\&.
+.RE
+.PP
+\-U, \-\-startup
+.RS 4
+Look for existing NVMe connections (i\&.e\&. transport addresses with at least one connected controller) during startup, start a discovery controller on every detected connection, retrieve the log page and (if
+\fI\-\-autoconnect\fR
+is also given) connect to all discovered controllers\&. Use the
+\fI\-\-persistent\fR
+option to make the Discovery Controllers persist after retrieving the Discovery Log page\&.
+.RE
+.PP
+\-p, \-\-persistent
+.RS 4
+This option has the same meaning as for
+\fInvme discover\fR; Discovery Controllers will remain after the Discovery Log Page has been retrieved\&. This is necessary for receiving AENs from the transport connection that the Discovery controller is associated with\&. Without this option, AENs can only be received and acted upon if the administrator manually creates persistent Discovery controller connections\&. When
+\fInvme monitor\fR
+terminates, it will attempt to take down all Discovery Controllers it has created (i\&.e\&. ones that didn\(cqt already exist when
+\fInvme monitor\fR
+was started\&.
+.RE
+.PP
+\-q <hostnqn>, \-\-hostnqn=<hostnqn>, \-I <hostid>, \-\-hostid=<hostid>, \-k <#>, \-\-keep\-alive\-tmo=<#>, \-c <#>, \-\-reconnect\-delay=<#>, \-l <#>, \-\-ctrl\-loss\-tmo=<#>, \-g, \-\-hdr\-digest, \-G, \-\-data\-digest, \-i <#>, \-\-nr\-io\-queues=<#>, \-W <#>, \-\-nr\-write\-queues=<#>, \-P <#>, \-\-nr\-poll\-queues=<#>, \-Q <#>, \-\-queue\-size=<#>, \-m, \-\-matching
+.RS 4
+These options have the same meaning as for
+\fInvme connect\-all\fR\&. See the man page nvme\-connect\-all(1) for details\&.
+.RE
+.PP
+\-p, \-\-persistent
+.RS 4
+Don\(cqt remove the discovery controller after retrieving the discovery log page\&.
+.RE
+.PP
+\-S, \-\-silent
+.RS 4
+Only print warnings and severe error messages\&. Do not log discoveries and newly created controllers\&.
+.RE
+.PP
+\-v, \-\-verbose
+.RS 4
+Log informational messages\&. This option overrides
+\fI\-\-silent\fR\&.
+.RE
+.PP
+\-D, \-\-debug
+.RS 4
+Log informational and debug messages\&. This option overrieds
+\fI\-\-silent\fR
+and
+\fI\-\-verbose\fR\&.
+.RE
+.SH "EXAMPLES"
+.sp
+.RS 4
+.ie n \{\
+\h'-04'\(bu\h'+03'\c
+.\}
+.el \{\
+.sp -1
+.IP \(bu 2.3
+.\}
+Listen to FC\-NVME events and AENs, creating persistent Discovery Controllers on the way, and automatically connect to all discovered controllers:
+.sp
+.if n \{\
+.RS 4
+.\}
+.nf
+# nvme discover \-\-autoconnect \-\-persistent
+.fi
+.if n \{\
+.RE
+.\}
+.RE
+.sp
+.RS 4
+.ie n \{\
+\h'-04'\(bu\h'+03'\c
+.\}
+.el \{\
+.sp -1
+.IP \(bu 2.3
+.\}
+Like the above, but print more log messages and try to set up Discovery Controllers on startup, and using a different host NQN:
+.sp
+.if n \{\
+.RS 4
+.\}
+.nf
+# nvme discover \-\-autoconnect \-\-persistent \-\-startup \-\-verbose \e
+\-\-hostqn=host1\-rogue\-nqn
+.fi
+.if n \{\
+.RE
+.\}
+.RE
+.SH "SEE ALSO"
+.sp
+nvme\-discover(1) nvme\-connect\-all(1)
+.SH "NVME"
+.sp
+Part of the nvme\-user suite
diff --git a/Documentation/nvme-monitor.html b/Documentation/nvme-monitor.html
new file mode 100644
index 0000000..c2b9a64
--- /dev/null
+++ b/Documentation/nvme-monitor.html
@@ -0,0 +1,1049 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
+    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
+<meta name="generator" content="AsciiDoc" />
+<title>nvme-monitor(1)</title>
+<style type="text/css">
+/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
+
+/* Default font. */
+body {
+  font-family: Georgia,serif;
+}
+
+/* Title font. */
+h1, h2, h3, h4, h5, h6,
+div.title, caption.title,
+thead, p.table.header,
+#toctitle,
+#author, #revnumber, #revdate, #revremark,
+#footer {
+  font-family: Arial,Helvetica,sans-serif;
+}
+
+body {
+  margin: 1em 5% 1em 5%;
+}
+
+a {
+  color: blue;
+  text-decoration: underline;
+}
+a:visited {
+  color: fuchsia;
+}
+
+em {
+  font-style: italic;
+  color: navy;
+}
+
+strong {
+  font-weight: bold;
+  color: #083194;
+}
+
+h1, h2, h3, h4, h5, h6 {
+  color: #527bbd;
+  margin-top: 1.2em;
+  margin-bottom: 0.5em;
+  line-height: 1.3;
+}
+
+h1, h2, h3 {
+  border-bottom: 2px solid silver;
+}
+h2 {
+  padding-top: 0.5em;
+}
+h3 {
+  float: left;
+}
+h3 + * {
+  clear: left;
+}
+h5 {
+  font-size: 1.0em;
+}
+
+div.sectionbody {
+  margin-left: 0;
+}
+
+hr {
+  border: 1px solid silver;
+}
+
+p {
+  margin-top: 0.5em;
+  margin-bottom: 0.5em;
+}
+
+ul, ol, li > p {
+  margin-top: 0;
+}
+ul > li     { color: #aaa; }
+ul > li > * { color: black; }
+
+.monospaced, code, pre {
+  font-family: "Courier New", Courier, monospace;
+  font-size: inherit;
+  color: navy;
+  padding: 0;
+  margin: 0;
+}
+pre {
+  white-space: pre-wrap;
+}
+
+#author {
+  color: #527bbd;
+  font-weight: bold;
+  font-size: 1.1em;
+}
+#email {
+}
+#revnumber, #revdate, #revremark {
+}
+
+#footer {
+  font-size: small;
+  border-top: 2px solid silver;
+  padding-top: 0.5em;
+  margin-top: 4.0em;
+}
+#footer-text {
+  float: left;
+  padding-bottom: 0.5em;
+}
+#footer-badges {
+  float: right;
+  padding-bottom: 0.5em;
+}
+
+#preamble {
+  margin-top: 1.5em;
+  margin-bottom: 1.5em;
+}
+div.imageblock, div.exampleblock, div.verseblock,
+div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
+div.admonitionblock {
+  margin-top: 1.0em;
+  margin-bottom: 1.5em;
+}
+div.admonitionblock {
+  margin-top: 2.0em;
+  margin-bottom: 2.0em;
+  margin-right: 10%;
+  color: #606060;
+}
+
+div.content { /* Block element content. */
+  padding: 0;
+}
+
+/* Block element titles. */
+div.title, caption.title {
+  color: #527bbd;
+  font-weight: bold;
+  text-align: left;
+  margin-top: 1.0em;
+  margin-bottom: 0.5em;
+}
+div.title + * {
+  margin-top: 0;
+}
+
+td div.title:first-child {
+  margin-top: 0.0em;
+}
+div.content div.title:first-child {
+  margin-top: 0.0em;
+}
+div.content + div.title {
+  margin-top: 0.0em;
+}
+
+div.sidebarblock > div.content {
+  background: #ffffee;
+  border: 1px solid #dddddd;
+  border-left: 4px solid #f0f0f0;
+  padding: 0.5em;
+}
+
+div.listingblock > div.content {
+  border: 1px solid #dddddd;
+  border-left: 5px solid #f0f0f0;
+  background: #f8f8f8;
+  padding: 0.5em;
+}
+
+div.quoteblock, div.verseblock {
+  padding-left: 1.0em;
+  margin-left: 1.0em;
+  margin-right: 10%;
+  border-left: 5px solid #f0f0f0;
+  color: #888;
+}
+
+div.quoteblock > div.attribution {
+  padding-top: 0.5em;
+  text-align: right;
+}
+
+div.verseblock > pre.content {
+  font-family: inherit;
+  font-size: inherit;
+}
+div.verseblock > div.attribution {
+  padding-top: 0.75em;
+  text-align: left;
+}
+/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
+div.verseblock + div.attribution {
+  text-align: left;
+}
+
+div.admonitionblock .icon {
+  vertical-align: top;
+  font-size: 1.1em;
+  font-weight: bold;
+  text-decoration: underline;
+  color: #527bbd;
+  padding-right: 0.5em;
+}
+div.admonitionblock td.content {
+  padding-left: 0.5em;
+  border-left: 3px solid #dddddd;
+}
+
+div.exampleblock > div.content {
+  border-left: 3px solid #dddddd;
+  padding-left: 0.5em;
+}
+
+div.imageblock div.content { padding-left: 0; }
+span.image img { border-style: none; vertical-align: text-bottom; }
+a.image:visited { color: white; }
+
+dl {
+  margin-top: 0.8em;
+  margin-bottom: 0.8em;
+}
+dt {
+  margin-top: 0.5em;
+  margin-bottom: 0;
+  font-style: normal;
+  color: navy;
+}
+dd > *:first-child {
+  margin-top: 0.1em;
+}
+
+ul, ol {
+    list-style-position: outside;
+}
+ol.arabic {
+  list-style-type: decimal;
+}
+ol.loweralpha {
+  list-style-type: lower-alpha;
+}
+ol.upperalpha {
+  list-style-type: upper-alpha;
+}
+ol.lowerroman {
+  list-style-type: lower-roman;
+}
+ol.upperroman {
+  list-style-type: upper-roman;
+}
+
+div.compact ul, div.compact ol,
+div.compact p, div.compact p,
+div.compact div, div.compact div {
+  margin-top: 0.1em;
+  margin-bottom: 0.1em;
+}
+
+tfoot {
+  font-weight: bold;
+}
+td > div.verse {
+  white-space: pre;
+}
+
+div.hdlist {
+  margin-top: 0.8em;
+  margin-bottom: 0.8em;
+}
+div.hdlist tr {
+  padding-bottom: 15px;
+}
+dt.hdlist1.strong, td.hdlist1.strong {
+  font-weight: bold;
+}
+td.hdlist1 {
+  vertical-align: top;
+  font-style: normal;
+  padding-right: 0.8em;
+  color: navy;
+}
+td.hdlist2 {
+  vertical-align: top;
+}
+div.hdlist.compact tr {
+  margin: 0;
+  padding-bottom: 0;
+}
+
+.comment {
+  background: yellow;
+}
+
+.footnote, .footnoteref {
+  font-size: 0.8em;
+}
+
+span.footnote, span.footnoteref {
+  vertical-align: super;
+}
+
+#footnotes {
+  margin: 20px 0 20px 0;
+  padding: 7px 0 0 0;
+}
+
+#footnotes div.footnote {
+  margin: 0 0 5px 0;
+}
+
+#footnotes hr {
+  border: none;
+  border-top: 1px solid silver;
+  height: 1px;
+  text-align: left;
+  margin-left: 0;
+  width: 20%;
+  min-width: 100px;
+}
+
+div.colist td {
+  padding-right: 0.5em;
+  padding-bottom: 0.3em;
+  vertical-align: top;
+}
+div.colist td img {
+  margin-top: 0.3em;
+}
+
+@media print {
+  #footer-badges { display: none; }
+}
+
+#toc {
+  margin-bottom: 2.5em;
+}
+
+#toctitle {
+  color: #527bbd;
+  font-size: 1.1em;
+  font-weight: bold;
+  margin-top: 1.0em;
+  margin-bottom: 0.1em;
+}
+
+div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
+  margin-top: 0;
+  margin-bottom: 0;
+}
+div.toclevel2 {
+  margin-left: 2em;
+  font-size: 0.9em;
+}
+div.toclevel3 {
+  margin-left: 4em;
+  font-size: 0.9em;
+}
+div.toclevel4 {
+  margin-left: 6em;
+  font-size: 0.9em;
+}
+
+span.aqua { color: aqua; }
+span.black { color: black; }
+span.blue { color: blue; }
+span.fuchsia { color: fuchsia; }
+span.gray { color: gray; }
+span.green { color: green; }
+span.lime { color: lime; }
+span.maroon { color: maroon; }
+span.navy { color: navy; }
+span.olive { color: olive; }
+span.purple { color: purple; }
+span.red { color: red; }
+span.silver { color: silver; }
+span.teal { color: teal; }
+span.white { color: white; }
+span.yellow { color: yellow; }
+
+span.aqua-background { background: aqua; }
+span.black-background { background: black; }
+span.blue-background { background: blue; }
+span.fuchsia-background { background: fuchsia; }
+span.gray-background { background: gray; }
+span.green-background { background: green; }
+span.lime-background { background: lime; }
+span.maroon-background { background: maroon; }
+span.navy-background { background: navy; }
+span.olive-background { background: olive; }
+span.purple-background { background: purple; }
+span.red-background { background: red; }
+span.silver-background { background: silver; }
+span.teal-background { background: teal; }
+span.white-background { background: white; }
+span.yellow-background { background: yellow; }
+
+span.big { font-size: 2em; }
+span.small { font-size: 0.6em; }
+
+span.underline { text-decoration: underline; }
+span.overline { text-decoration: overline; }
+span.line-through { text-decoration: line-through; }
+
+div.unbreakable { page-break-inside: avoid; }
+
+
+/*
+ * xhtml11 specific
+ *
+ * */
+
+div.tableblock {
+  margin-top: 1.0em;
+  margin-bottom: 1.5em;
+}
+div.tableblock > table {
+  border: 3px solid #527bbd;
+}
+thead, p.table.header {
+  font-weight: bold;
+  color: #527bbd;
+}
+p.table {
+  margin-top: 0;
+}
+/* Because the table frame attribute is overridden by CSS in most browsers. */
+div.tableblock > table[frame="void"] {
+  border-style: none;
+}
+div.tableblock > table[frame="hsides"] {
+  border-left-style: none;
+  border-right-style: none;
+}
+div.tableblock > table[frame="vsides"] {
+  border-top-style: none;
+  border-bottom-style: none;
+}
+
+
+/*
+ * html5 specific
+ *
+ * */
+
+table.tableblock {
+  margin-top: 1.0em;
+  margin-bottom: 1.5em;
+}
+thead, p.tableblock.header {
+  font-weight: bold;
+  color: #527bbd;
+}
+p.tableblock {
+  margin-top: 0;
+}
+table.tableblock {
+  border-width: 3px;
+  border-spacing: 0px;
+  border-style: solid;
+  border-color: #527bbd;
+  border-collapse: collapse;
+}
+th.tableblock, td.tableblock {
+  border-width: 1px;
+  padding: 4px;
+  border-style: solid;
+  border-color: #527bbd;
+}
+
+table.tableblock.frame-topbot {
+  border-left-style: hidden;
+  border-right-style: hidden;
+}
+table.tableblock.frame-sides {
+  border-top-style: hidden;
+  border-bottom-style: hidden;
+}
+table.tableblock.frame-none {
+  border-style: hidden;
+}
+
+th.tableblock.halign-left, td.tableblock.halign-left {
+  text-align: left;
+}
+th.tableblock.halign-center, td.tableblock.halign-center {
+  text-align: center;
+}
+th.tableblock.halign-right, td.tableblock.halign-right {
+  text-align: right;
+}
+
+th.tableblock.valign-top, td.tableblock.valign-top {
+  vertical-align: top;
+}
+th.tableblock.valign-middle, td.tableblock.valign-middle {
+  vertical-align: middle;
+}
+th.tableblock.valign-bottom, td.tableblock.valign-bottom {
+  vertical-align: bottom;
+}
+
+
+/*
+ * manpage specific
+ *
+ * */
+
+body.manpage h1 {
+  padding-top: 0.5em;
+  padding-bottom: 0.5em;
+  border-top: 2px solid silver;
+  border-bottom: 2px solid silver;
+}
+body.manpage h2 {
+  border-style: none;
+}
+body.manpage div.sectionbody {
+  margin-left: 3em;
+}
+
+@media print {
+  body.manpage div#toc { display: none; }
+}
+
+
+</style>
+<script type="text/javascript">
+/*<![CDATA[*/
+var asciidoc = {  // Namespace.
+
+/////////////////////////////////////////////////////////////////////
+// Table Of Contents generator
+/////////////////////////////////////////////////////////////////////
+
+/* Author: Mihai Bazon, September 2002
+ * http://students.infoiasi.ro/~mishoo
+ *
+ * Table Of Content generator
+ * Version: 0.4
+ *
+ * Feel free to use this script under the terms of the GNU General Public
+ * License, as long as you do not remove or alter this notice.
+ */
+
+ /* modified by Troy D. Hanson, September 2006. License: GPL */
+ /* modified by Stuart Rackham, 2006, 2009. License: GPL */
+
+// toclevels = 1..4.
+toc: function (toclevels) {
+
+  function getText(el) {
+    var text = "";
+    for (var i = el.firstChild; i != null; i = i.nextSibling) {
+      if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
+        text += i.data;
+      else if (i.firstChild != null)
+        text += getText(i);
+    }
+    return text;
+  }
+
+  function TocEntry(el, text, toclevel) {
+    this.element = el;
+    this.text = text;
+    this.toclevel = toclevel;
+  }
+
+  function tocEntries(el, toclevels) {
+    var result = new Array;
+    var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
+    // Function that scans the DOM tree for header elements (the DOM2
+    // nodeIterator API would be a better technique but not supported by all
+    // browsers).
+    var iterate = function (el) {
+      for (var i = el.firstChild; i != null; i = i.nextSibling) {
+        if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
+          var mo = re.exec(i.tagName);
+          if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
+            result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
+          }
+          iterate(i);
+        }
+      }
+    }
+    iterate(el);
+    return result;
+  }
+
+  var toc = document.getElementById("toc");
+  if (!toc) {
+    return;
+  }
+
+  // Delete existing TOC entries in case we're reloading the TOC.
+  var tocEntriesToRemove = [];
+  var i;
+  for (i = 0; i < toc.childNodes.length; i++) {
+    var entry = toc.childNodes[i];
+    if (entry.nodeName.toLowerCase() == 'div'
+     && entry.getAttribute("class")
+     && entry.getAttribute("class").match(/^toclevel/))
+      tocEntriesToRemove.push(entry);
+  }
+  for (i = 0; i < tocEntriesToRemove.length; i++) {
+    toc.removeChild(tocEntriesToRemove[i]);
+  }
+
+  // Rebuild TOC entries.
+  var entries = tocEntries(document.getElementById("content"), toclevels);
+  for (var i = 0; i < entries.length; ++i) {
+    var entry = entries[i];
+    if (entry.element.id == "")
+      entry.element.id = "_toc_" + i;
+    var a = document.createElement("a");
+    a.href = "#" + entry.element.id;
+    a.appendChild(document.createTextNode(entry.text));
+    var div = document.createElement("div");
+    div.appendChild(a);
+    div.className = "toclevel" + entry.toclevel;
+    toc.appendChild(div);
+  }
+  if (entries.length == 0)
+    toc.parentNode.removeChild(toc);
+},
+
+
+/////////////////////////////////////////////////////////////////////
+// Footnotes generator
+/////////////////////////////////////////////////////////////////////
+
+/* Based on footnote generation code from:
+ * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
+ */
+
+footnotes: function () {
+  // Delete existing footnote entries in case we're reloading the footnodes.
+  var i;
+  var noteholder = document.getElementById("footnotes");
+  if (!noteholder) {
+    return;
+  }
+  var entriesToRemove = [];
+  for (i = 0; i < noteholder.childNodes.length; i++) {
+    var entry = noteholder.childNodes[i];
+    if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
+      entriesToRemove.push(entry);
+  }
+  for (i = 0; i < entriesToRemove.length; i++) {
+    noteholder.removeChild(entriesToRemove[i]);
+  }
+
+  // Rebuild footnote entries.
+  var cont = document.getElementById("content");
+  var spans = cont.getElementsByTagName("span");
+  var refs = {};
+  var n = 0;
+  for (i=0; i<spans.length; i++) {
+    if (spans[i].className == "footnote") {
+      n++;
+      var note = spans[i].getAttribute("data-note");
+      if (!note) {
+        // Use [\s\S] in place of . so multi-line matches work.
+        // Because JavaScript has no s (dotall) regex flag.
+        note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
+        spans[i].innerHTML =
+          "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
+          "' title='View footnote' class='footnote'>" + n + "</a>]";
+        spans[i].setAttribute("data-note", note);
+      }
+      noteholder.innerHTML +=
+        "<div class='footnote' id='_footnote_" + n + "'>" +
+        "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
+        n + "</a>. " + note + "</div>";
+      var id =spans[i].getAttribute("id");
+      if (id != null) refs["#"+id] = n;
+    }
+  }
+  if (n == 0)
+    noteholder.parentNode.removeChild(noteholder);
+  else {
+    // Process footnoterefs.
+    for (i=0; i<spans.length; i++) {
+      if (spans[i].className == "footnoteref") {
+        var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
+        href = href.match(/#.*/)[0];  // Because IE return full URL.
+        n = refs[href];
+        spans[i].innerHTML =
+          "[<a href='#_footnote_" + n +
+          "' title='View footnote' class='footnote'>" + n + "</a>]";
+      }
+    }
+  }
+},
+
+install: function(toclevels) {
+  var timerId;
+
+  function reinstall() {
+    asciidoc.footnotes();
+    if (toclevels) {
+      asciidoc.toc(toclevels);
+    }
+  }
+
+  function reinstallAndRemoveTimer() {
+    clearInterval(timerId);
+    reinstall();
+  }
+
+  timerId = setInterval(reinstall, 500);
+  if (document.addEventListener)
+    document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
+  else
+    window.onload = reinstallAndRemoveTimer;
+}
+
+}
+asciidoc.install();
+/*]]>*/
+</script>
+</head>
+<body class="manpage">
+<div id="header">
+<h1>
+nvme-monitor(1) Manual Page
+</h1>
+<h2>NAME</h2>
+<div class="sectionbody">
+<p>nvme-monitor -
+   Monitor Discovery events and Discover and Connect automatically
+</p>
+</div>
+</div>
+<div id="content">
+<div class="sect1">
+<h2 id="_synopsis">SYNOPSIS</h2>
+<div class="sectionbody">
+<div class="verseblock">
+<pre class="content"><em>nvme discover</em>
+                [--autoconnect            | -A
+                [--startup                | -U
+                [--persistent             | -p
+                [--hostnqn=&lt;hostnqn&gt;      | -q &lt;hostnqn&gt;]
+                [--hostid=&lt;hostid&gt;        | -I &lt;hostid&gt;]
+                [--keep-alive-tmo=&lt;#&gt;     | -k &lt;#&gt;]
+                [--reconnect-delay=&lt;#&gt;    | -c &lt;#&gt;]
+                [--ctrl-loss-tmo=&lt;#&gt;      | -l &lt;#&gt;]
+                [--hdr-digest             | -g]
+                [--data-digest            | -G]
+                [--nr-io-queues=&lt;#&gt;       | -i &lt;#&gt;]
+                [--nr-write-queues=&lt;#&gt;    | -W &lt;#&gt;]
+                [--nr-poll-queues=&lt;#&gt;     | -P &lt;#&gt;]
+                [--queue-size=&lt;#&gt;         | -Q &lt;#&gt;]
+                [--matching               | -m]
+                [--persistent             | -p]
+                [--silent                 | -S]
+                [--verbose                | -v]
+                [--debug                  | -D]
+                [--clockstamps            | -C]</pre>
+<div class="attribution">
+</div></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_description">DESCRIPTION</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Listen to Discovery events (Asynchronous Event Notifications, AENs) on
+NVMe-over-Fabrics (NVMeoF) Discovery Controllers and for other events related
+to NVMeoF Discovery, and optionally connect to newly discovered controllers.</p></div>
+<div class="paragraph"><p>If no parameters are given, then <em>nvme monitor</em> listens to Discovery-related
+udev events (uevents). If an event is received, it connects to the Discovery
+Controller and performs the equivalent of an <em>nvme discover</em> on the associated
+transport address. If the <em>--autoconnect</em> option is given, it performs the
+equivalent of an <em>nvme connect-all</em> instead. When run through a systemd
+service in <em>--autoconnect</em> mode, the monitor can be used as an alternative to
+the udev-rule based auto-activation of NVMeoF connections.</p></div>
+<div class="paragraph"><p>Currently, the following event types are supported:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+"Discovery Log Page Change" Asynchronous Event Notifications (AENs)
+  delivered via persistent connections to NVMeoF discovery controllers
+  connected to the discovery service (<code>nqn.2014-08.org.nvmexpress.discovery</code>).
+  Note that without the <em>--persistent</em> option, <em>nvme monitor</em> will not create
+  persistent Discovery controllers itself, but it will receive and act upon
+  AENs sent from persistent Discovery Controllers which were created manually
+  or by other programs.
+</p>
+</li>
+<li>
+<p>
+FC-NVMe auto-connect uevents sent when the FibreChannel transport discovers
+  N_Ports offering NVMe services.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>See the documentation for the nvme-connect-all(1) and nvme-discover(1)
+commands for further background.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_options">OPTIONS</h2>
+<div class="sectionbody">
+<div class="dlist"><dl>
+<dt class="hdlist1">
+-A
+</dt>
+<dt class="hdlist1">
+--autoconnect
+</dt>
+<dd>
+<p>
+        If this option is given, <em>nvme monitor</em> will attempt to connect to
+        newly discovered controllers.
+        In order to avoid race conditions with <em>nvme connect-all</em> commands
+        spawned by udev rules via systemd&#8217;s <code>nvmf-connect@.service</code>, an empty
+        rule is created under <code>/run/udev/rules.d/70-nvmf-autoconnect.rules</code>,
+        disabling the udev rules that start this service. The file will be
+        deleted when <em>nvme monitor</em> terminates.
+</p>
+</dd>
+<dt class="hdlist1">
+-U
+</dt>
+<dt class="hdlist1">
+--startup
+</dt>
+<dd>
+<p>
+        Look for existing NVMe connections (i.e. transport addresses with at
+        least one connected controller) during startup, start a discovery
+        controller on every detected connection, retrieve the log page and (if
+        <em>--autoconnect</em> is also given) connect to all discovered controllers.
+        Use the <em>--persistent</em> option to make the Discovery Controllers
+        persist after retrieving the Discovery Log page.
+</p>
+</dd>
+<dt class="hdlist1">
+-p
+</dt>
+<dt class="hdlist1">
+--persistent
+</dt>
+<dd>
+<p>
+        This option has the same meaning as for <em>nvme discover</em>; Discovery
+        Controllers will remain after the Discovery Log Page has been
+        retrieved. This is necessary for receiving AENs from the transport
+        connection that the Discovery controller is associated with. Without
+        this option, AENs can only be received and acted upon if the
+        administrator manually creates persistent Discovery controller
+        connections.
+        When <em>nvme monitor</em> terminates, it will attempt to take down all
+        Discovery Controllers it has created (i.e. ones that didn&#8217;t already
+        exist when <em>nvme monitor</em> was started.
+</p>
+</dd>
+<dt class="hdlist1">
+-q &lt;hostnqn&gt;
+</dt>
+<dt class="hdlist1">
+--hostnqn=&lt;hostnqn&gt;
+</dt>
+<dt class="hdlist1">
+-I &lt;hostid&gt;
+</dt>
+<dt class="hdlist1">
+--hostid=&lt;hostid&gt;
+</dt>
+<dt class="hdlist1">
+-k &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--keep-alive-tmo=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-c &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--reconnect-delay=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-l &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--ctrl-loss-tmo=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-g
+</dt>
+<dt class="hdlist1">
+--hdr-digest
+</dt>
+<dt class="hdlist1">
+-G
+</dt>
+<dt class="hdlist1">
+--data-digest
+</dt>
+<dt class="hdlist1">
+-i &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--nr-io-queues=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-W &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--nr-write-queues=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-P &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--nr-poll-queues=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-Q &lt;#&gt;
+</dt>
+<dt class="hdlist1">
+--queue-size=&lt;#&gt;
+</dt>
+<dt class="hdlist1">
+-m
+</dt>
+<dt class="hdlist1">
+--matching
+</dt>
+<dd>
+<p>
+        These options have the same meaning as for <em>nvme connect-all</em>. See the
+        man page nvme-connect-all(1) for details.
+</p>
+</dd>
+<dt class="hdlist1">
+-p
+</dt>
+<dt class="hdlist1">
+--persistent
+</dt>
+<dd>
+<p>
+        Don&#8217;t remove the discovery controller after retrieving the discovery
+        log page.
+</p>
+</dd>
+<dt class="hdlist1">
+-S
+</dt>
+<dt class="hdlist1">
+--silent
+</dt>
+<dd>
+<p>
+        Only print warnings and severe error messages. Do not log discoveries
+        and newly created controllers.
+</p>
+</dd>
+<dt class="hdlist1">
+-v
+</dt>
+<dt class="hdlist1">
+--verbose
+</dt>
+<dd>
+<p>
+        Log informational messages. This option overrides <em>--silent</em>.
+</p>
+</dd>
+<dt class="hdlist1">
+-D
+</dt>
+<dt class="hdlist1">
+--debug
+</dt>
+<dd>
+<p>
+        Log informational and debug messages. This option overrieds <em>--silent</em>
+        and <em>--verbose</em>.
+</p>
+</dd>
+</dl></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_examples">EXAMPLES</h2>
+<div class="sectionbody">
+<div class="ulist"><ul>
+<li>
+<p>
+Listen to FC-NVME events and AENs, creating persistent Discovery Controllers
+on the way, and automatically connect to all discovered controllers:
+</p>
+<div class="listingblock">
+<div class="content">
+<pre><code># nvme discover --autoconnect --persistent</code></pre>
+</div></div>
+</li>
+<li>
+<p>
+Like the above, but print more log messages and try to set up Discovery
+Controllers on startup, and using a different host NQN:
+</p>
+<div class="listingblock">
+<div class="content">
+<pre><code># nvme discover --autoconnect --persistent --startup --verbose \
+--hostqn=host1-rogue-nqn</code></pre>
+</div></div>
+</li>
+</ul></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_see_also">SEE ALSO</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>nvme-discover(1)
+nvme-connect-all(1)</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_nvme">NVME</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Part of the nvme-user suite</p></div>
+</div>
+</div>
+</div>
+<div id="footnotes"><hr /></div>
+<div id="footer">
+<div id="footer-text">
+Last updated
+ 2021-01-20 23:53:52 CET
+</div>
+</div>
+</body>
+</html>
diff --git a/Documentation/nvme-monitor.txt b/Documentation/nvme-monitor.txt
new file mode 100644
index 0000000..9232fab
--- /dev/null
+++ b/Documentation/nvme-monitor.txt
@@ -0,0 +1,169 @@
+nvme-monitor(1)
+===============
+
+NAME
+----
+nvme-monitor - Monitor Discovery events and Discover and Connect automatically
+
+SYNOPSIS
+--------
+[verse]
+'nvme discover'
+		[--autoconnect            | -A]
+		[--startup                | -U]
+		[--persistent             | -p]
+		[--hostnqn=<hostnqn>      | -q <hostnqn>]
+		[--hostid=<hostid>        | -I <hostid>]
+		[--keep-alive-tmo=<#>     | -k <#>]
+		[--reconnect-delay=<#>    | -c <#>]
+		[--ctrl-loss-tmo=<#>      | -l <#>]
+		[--hdr-digest             | -g]
+		[--data-digest            | -G]
+		[--nr-io-queues=<#>       | -i <#>]
+		[--nr-write-queues=<#>    | -W <#>]
+		[--nr-poll-queues=<#>     | -P <#>]
+		[--queue-size=<#>         | -Q <#>]
+		[--matching               | -m]
+		[--persistent             | -p]
+		[--silent                 | -S]
+		[--verbose                | -v]
+		[--debug                  | -D]
+		[--clockstamps            | -C]
+
+DESCRIPTION
+-----------
+Listen to Discovery events (Asynchronous Event Notifications, AENs) on
+NVMe-over-Fabrics (NVMeoF) Discovery Controllers and for other events related
+to NVMeoF Discovery, and optionally connect to newly discovered controllers.
+
+If no parameters are given, then 'nvme monitor' listens to Discovery-related
+udev events (uevents). If an event is received, it connects to the Discovery
+Controller and performs the equivalent of an 'nvme discover' on the associated
+transport address. If the '--autoconnect' option is given, it performs the
+equivalent of an 'nvme connect-all' instead. When run through a systemd
+service in '--autoconnect' mode, the monitor can be used as an alternative to
+the udev-rule based auto-activation of NVMeoF connections.
+
+Currently, the following event types are supported:
+
+- "Discovery Log Page Change" Asynchronous Event Notifications (AENs)
+  delivered via persistent connections to NVMeoF discovery controllers
+  connected to the discovery service (`nqn.2014-08.org.nvmexpress.discovery`).
+  Note that without the '--persistent' option, 'nvme monitor' will not create
+  persistent Discovery controllers itself, but it will receive and act upon
+  AENs sent from persistent Discovery Controllers which were created manually
+  or by other programs.
+
+- FC-NVMe auto-connect uevents sent when the FibreChannel transport discovers
+  N_Ports offering NVMe services.
+
+See the documentation for the nvme-connect-all(1) and nvme-discover(1)
+commands for further background.
+
+OPTIONS
+-------
+
+-A::
+--autoconnect::
+	If this option is given, 'nvme monitor' will attempt to connect to
+	newly discovered controllers.
+	In order to avoid race conditions with 'nvme connect-all' commands
+	spawned by udev rules via systemd's `nvmf-connect@.service`, an empty
+	rule is created under `/run/udev/rules.d/70-nvmf-autoconnect.rules`,
+	disabling the udev rules that start this service. The file will be
+	deleted when 'nvme monitor' terminates.
+	
+-U::
+--startup::
+	Look for existing NVMe connections (i.e. transport addresses with at
+	least one connected controller) during startup, start a discovery
+	controller on every detected connection, retrieve the log page and (if
+	'--autoconnect' is also given) connect to all discovered controllers.
+	Use the '--persistent' option to make the Discovery Controllers
+	persist after retrieving the Discovery Log page.
+
+-p::
+--persistent::
+	This option has the same meaning as for 'nvme discover'; Discovery
+	Controllers will remain after the Discovery Log Page has been
+	retrieved. This is necessary for receiving AENs from the transport
+	connection that the Discovery controller is associated with. Without
+	this option, AENs can only be received and acted upon if the
+	administrator manually creates persistent Discovery controller
+	connections.
+	When 'nvme monitor' terminates, it will attempt to take down all
+	Discovery Controllers it has created (i.e. ones that didn't already
+	exist when 'nvme monitor' was started.
+	
+-q <hostnqn>::
+--hostnqn=<hostnqn>::
+-I <hostid>::
+--hostid=<hostid>::
+-k <#>::
+--keep-alive-tmo=<#>::
+-c <#>::
+--reconnect-delay=<#>::
+-l <#>::
+--ctrl-loss-tmo=<#>::
+-g::
+--hdr-digest::
+-G::
+--data-digest::
+-i <#>::
+--nr-io-queues=<#>::
+-W <#>::
+--nr-write-queues=<#>::
+-P <#>::
+--nr-poll-queues=<#>::
+-Q <#>::
+--queue-size=<#>::
+-m::
+--matching::
+	These options have the same meaning as for 'nvme connect-all'. See the
+	man page nvme-connect-all(1) for details.
+
+-p::
+--persistent::
+	Don't remove the discovery controller after retrieving the discovery
+	log page.
+
+-S::
+--silent::
+	Only print warnings and severe error messages. Do not log discoveries
+	and newly created controllers.
+
+-v::
+--verbose::
+	Log informational messages. This option overrides '--silent'.
+
+-D::
+--debug::
+	Log informational and debug messages. This option overrieds '--silent'
+	and '--verbose'.
+
+
+EXAMPLES
+--------
+* Listen to FC-NVME events and AENs, creating persistent Discovery Controllers
+on the way, and automatically connect to all discovered controllers:
++
+------------
+# nvme discover --autoconnect --persistent
+------------
++
+* Like the above, but print more log messages and try to set up Discovery
+Controllers on startup, and using a different host NQN:
++
+-----------
+# nvme discover --autoconnect --persistent --startup --verbose \
+--hostqn=host1-rogue-nqn
+------------
+
+SEE ALSO
+--------
+nvme-discover(1)
+nvme-connect-all(1)
+
+NVME
+----
+Part of the nvme-user suite
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 35/35] monitor: add option --keep/-K

[mwilck]

From: Martin Wilck <mwilck@suse.com>

With this option, "nvme monitor" will keep created discovery controllers
running even after it exits. If running in the initrd, this is will be
done automatically. That has the advantage that the new monitor instance
that will be started automatically after switching root can simply take
over the already created discovery controllers.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 Documentation/nvme-monitor.1    | 28 +++++++++++++++++++++-------
 Documentation/nvme-monitor.html | 30 ++++++++++++++++++++++++------
 Documentation/nvme-monitor.txt  |  1 +
 monitor.c                       | 15 ++++++++++++++-
 4 files changed, 60 insertions(+), 14 deletions(-)

diff --git a/Documentation/nvme-monitor.1 b/Documentation/nvme-monitor.1
index 1578e4c..36b015d 100644
--- a/Documentation/nvme-monitor.1
+++ b/Documentation/nvme-monitor.1
@@ -2,12 +2,12 @@
 .\"     Title: nvme-monitor
 .\"    Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author]
 .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/>
-.\"      Date: 01/20/2021
+.\"      Date: 01/25/2021
 .\"    Manual: NVMe Manual
 .\"    Source: NVMe
 .\"  Language: English
 .\"
-.TH "NVME\-MONITOR" "1" "01/20/2021" "NVMe" "NVMe Manual"
+.TH "NVME\-MONITOR" "1" "01/25/2021" "NVMe" "NVMe Manual"
 .\" -----------------------------------------------------------------
 .\" * Define some portability stuff
 .\" -----------------------------------------------------------------
@@ -33,9 +33,10 @@ nvme-monitor \- Monitor Discovery events and Discover and Connect automatically
 .sp
 .nf
 \fInvme discover\fR
-                [\-\-autoconnect            | \-A
-                [\-\-startup                | \-U
-                [\-\-persistent             | \-p
+                [\-\-autoconnect            | \-A]
+                [\-\-startup                | \-U]
+                [\-\-persistent             | \-p]
+                [\-\-keep                   | \-K]
                 [\-\-hostnqn=<hostnqn>      | \-q <hostnqn>]
                 [\-\-hostid=<hostid>        | \-I <hostid>]
                 [\-\-keep\-alive\-tmo=<#>     | \-k <#>]
@@ -118,9 +119,22 @@ option to make the Discovery Controllers persist after retrieving the Discovery
 This option has the same meaning as for
 \fInvme discover\fR; Discovery Controllers will remain after the Discovery Log Page has been retrieved\&. This is necessary for receiving AENs from the transport connection that the Discovery controller is associated with\&. Without this option, AENs can only be received and acted upon if the administrator manually creates persistent Discovery controller connections\&. When
 \fInvme monitor\fR
-terminates, it will attempt to take down all Discovery Controllers it has created (i\&.e\&. ones that didn\(cqt already exist when
+terminates and the, it will attempt to take down all Discovery Controllers it has created (i\&.e\&. ones that didn\(cqt already exist when
 \fInvme monitor\fR
-was started\&.
+was started (but see
+\fI\-\-keep\fR)\&.
+.RE
+.PP
+\-K, \-\-keep
+.RS 4
+Do not tear down created persistent discovery controllers when exiting\&. This option makes only sense together with
+\fI\-\-persistent\fR\&. When
+\fInvme monitor\fR
+detects that it is running in the initial RAM FS (initrd),
+\fI\-\-keep\fR
+is implicitly assumed, i\&.e\&. discovery connections are not stopped\&. This is because after switching from the initrd to the real root file system, another
+\fInvme monitor\fR
+instance will probably be started that can reuse the created controller instances\&.
 .RE
 .PP
 \-q <hostnqn>, \-\-hostnqn=<hostnqn>, \-I <hostid>, \-\-hostid=<hostid>, \-k <#>, \-\-keep\-alive\-tmo=<#>, \-c <#>, \-\-reconnect\-delay=<#>, \-l <#>, \-\-ctrl\-loss\-tmo=<#>, \-g, \-\-hdr\-digest, \-G, \-\-data\-digest, \-i <#>, \-\-nr\-io\-queues=<#>, \-W <#>, \-\-nr\-write\-queues=<#>, \-P <#>, \-\-nr\-poll\-queues=<#>, \-Q <#>, \-\-queue\-size=<#>, \-m, \-\-matching
diff --git a/Documentation/nvme-monitor.html b/Documentation/nvme-monitor.html
index c2b9a64..bd57a22 100644
--- a/Documentation/nvme-monitor.html
+++ b/Documentation/nvme-monitor.html
@@ -750,9 +750,10 @@ nvme-monitor(1) Manual Page
 <div class="sectionbody">
 <div class="verseblock">
 <pre class="content"><em>nvme discover</em>
-                [--autoconnect            | -A
-                [--startup                | -U
-                [--persistent             | -p
+                [--autoconnect            | -A]
+                [--startup                | -U]
+                [--persistent             | -p]
+                [--keep                   | -K]
                 [--hostnqn=&lt;hostnqn&gt;      | -q &lt;hostnqn&gt;]
                 [--hostid=&lt;hostid&gt;        | -I &lt;hostid&gt;]
                 [--keep-alive-tmo=&lt;#&gt;     | -k &lt;#&gt;]
@@ -863,9 +864,26 @@ commands for further background.</p></div>
         this option, AENs can only be received and acted upon if the
         administrator manually creates persistent Discovery controller
         connections.
-        When <em>nvme monitor</em> terminates, it will attempt to take down all
+        When <em>nvme monitor</em> terminates and the, it will attempt to take down all
         Discovery Controllers it has created (i.e. ones that didn&#8217;t already
-        exist when <em>nvme monitor</em> was started.
+        exist when <em>nvme monitor</em> was started (but see <em>--keep</em>).
+</p>
+</dd>
+<dt class="hdlist1">
+-K
+</dt>
+<dt class="hdlist1">
+--keep
+</dt>
+<dd>
+<p>
+        Do not tear down created persistent discovery controllers when
+        exiting. This option makes only sense together with <em>--persistent</em>.
+        When <em>nvme monitor</em> detects that it is running in the initial RAM FS
+        (initrd), <em>--keep</em> is implicitly assumed, i.e. discovery connections
+        are not stopped. This is because after switching from the initrd to
+        the real root file system, another <em>nvme monitor</em> instance will
+        probably be started that can reuse the created controller instances.
 </p>
 </dd>
 <dt class="hdlist1">
@@ -1042,7 +1060,7 @@ nvme-connect-all(1)</p></div>
 <div id="footer">
 <div id="footer-text">
 Last updated
- 2021-01-20 23:53:52 CET
+ 2021-01-25 22:35:29 CET
 </div>
 </div>
 </body>
diff --git a/Documentation/nvme-monitor.txt b/Documentation/nvme-monitor.txt
index 9232fab..517d036 100644
--- a/Documentation/nvme-monitor.txt
+++ b/Documentation/nvme-monitor.txt
@@ -12,6 +12,7 @@ SYNOPSIS
 		[--autoconnect            | -A]
 		[--startup                | -U]
 		[--persistent             | -p]
+		[--keep                   | -K]
 		[--hostnqn=<hostnqn>      | -q <hostnqn>]
 		[--hostid=<hostid>        | -I <hostid>]
 		[--keep-alive-tmo=<#>     | -k <#>]
diff --git a/monitor.c b/monitor.c
index ecf6706..40d1f29 100644
--- a/monitor.c
+++ b/monitor.c
@@ -44,6 +44,7 @@ static struct monitor_config {
 	bool autoconnect;
 	bool skip_udev_on_exit;
 	bool start_ctrls;
+	bool keep_ctrls;
 } mon_cfg;
 
 static struct udev *udev;
@@ -669,6 +670,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
 		OPT_FLAG("startup",        'U', &mon_cfg.start_ctrls, "start discovery controllers on startup"),
 		OPT_FLAG("persistent",     'p', &cfg.persistent,      "persistent discovery connections"),
+		OPT_FLAG("keep",           'K', &mon_cfg.keep_ctrls,  "keep discovery controllers after exit"),
 		OPT_LIST("hostnqn",        'q', &cfg.hostnqn,         "user-defined hostnqn (if default not used)"),
 		OPT_LIST("hostid",         'I', &cfg.hostid,          "user-defined hostid (if default not used)"),
 		OPT_INT("keep-alive-tmo",  'k', &cfg.keep_alive_tmo,  "keep alive timeout period in seconds"),
@@ -716,6 +718,11 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
 	return ret;
 }
 
+static bool running_in_initrd(void)
+{
+	return access("/etc/initrd-release", F_OK) >= 0;
+}
+
 int aen_monitor(const char *desc, int argc, char **argv)
 {
 	int ret;
@@ -724,11 +731,16 @@ int aen_monitor(const char *desc, int argc, char **argv)
 	ret = monitor_parse_opts(desc, argc, argv);
 	if (ret)
 		goto out;
+
 	ret = monitor_init_signals();
 	if (ret != 0) {
 		log(LOG_ERR, "monitor: failed to initialize signals: %m\n");
 		goto out;
 	}
+
+	if (running_in_initrd())
+		mon_cfg.keep_ctrls = true;
+
 	ret = create_udev_monitor(&monitor);
 	if (ret == 0) {
 		conndb_init_from_sysfs();
@@ -739,7 +751,8 @@ int aen_monitor(const char *desc, int argc, char **argv)
 	}
 
 	conndb_for_each(monitor_kill_discovery_task, NULL);
-	if (mon_cfg.autoconnect || mon_cfg.start_ctrls)
+	if ((mon_cfg.autoconnect || mon_cfg.start_ctrls) &&
+	    !mon_cfg.keep_ctrls)
 		conndb_for_each(monitor_remove_discovery_ctrl, NULL);
 	conndb_free();
 
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Sagi Grimberg]

> From: Martin Wilck <mwilck@suse.com>
> 
> (Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)
> 
> This patch set adds a new subcommand **nvme monitor**. In this mode,
> **nvme-cli** runs continuously, monitors events (currently, uevents) relevant
> for discovery, and optionally autoconnects to newly discovered subsystems.
> 
> The monitor mode is suitable to be run in a systemd service. An appropriate
> unit file is provided. As such, **nvme monitor** can be used as an alternative
> to the current auto-connection mechanism based on udev rules and systemd
> template units. If `--autoconnect` is active, **nvme monitor** masks the
> respective udev rules in order to prevent simultaneous connection attempts
> from udev and itself.

I think that a monitor daemon is a good path forward.

> 
> This method for discovery and autodetection has some advantages over the
> current udev-rule based approach:
> 
>   * By using the `--persistent` option, users can easily control whether
>     persistent discovery controllers for discovered transport addresses should
>     be created and monitored for AEN events. **nvme monitor** watches known
>     transport addresses, creates discovery controllers as required, and re-uses
>     existing ones if possible.

What does that mean?

  It keeps track of persistent discovery
>     controllers it created, and tears them down on exit. When run in
>     `--persistent --autoconnect` mode *in the initial ramfs*, it will keep
>     discovery controllers alive, so that a new instance started after switching
>     root can simply re-use them.

Nice.

>   * In certain situations, the systemd-based approach may miss events due to
>     race conditions. This can happen e.g. if an FC remote port is detected, but
>     shortly after it's detection an FC relogin procedure is necessary e.g. due to
>     an RSCN. In this case, an `fc_udev_device` uevent will be received on the
>     first detection and handled by an `nvme connect-all` command run from
>     `nvmf-connect@.service`. The connection attempt to the rport in question will
>     fail with "no such device" because of the simultaneous FC
>     relogin. `nvmf-connect@.service` may not terminate immediately, because it
>     attempts to establish other connections listed in the Discovery Log page it
>     retrieved. When the FC relogin eventually finishes, a new uevent will be
>     received, and `nvmf-connect@` will be started again, but *this has no effect*
>     if the previous `nvmf-connect@` service hasn't finished yet. This is the
>     general semantics of systemd services, no easy workaround exists.  **nvme
>     monitor** doesn't suffer from this problem. If it sees an uevent for a
>     transport address for which a discovery is already running, it will queue the
>     handling of this event up and restart the discovery after it's finished.

While I understand the issue, this reason alone is an overkill for doing 
this.

>   * Resource consumption for handling uevents is lower. Instead of running an
>     udev worker, executing the rules, executing `systemctl start` from the
>     worker, starting a systemd service, and starting a separate **nvme-cli**
>     instance, only a single `fork()` operation is necessary. Of course, on the
>     back side, the monitor itself consumes resources while it's running and
>     waiting for events. On my system with 8 persistent discovery controllers,
>     its RSS is ~3MB. CPU consumption is zero as long as no events occur.

What is the baseline with what we have today?

>   * **nvme monitor** could be easily extended to handle events for non-FC
>     transports.

Which events?

> 
> I've tested `fc_udev_device` handling for NVMeoFC with an Ontap target, and
> AEN handling for RDMA using a Linux **nvmet** target.
> 
> ### Implementation notes
> 
> I've tried to change the exisiting **nvme-cli** code as little as possible
> while reusing the code from `fabrics.c`. The majority of changes in the
> existing code exports formerly static functions and variables, so that they
> are usable from the monitor code.

General comment, can you please separate out fixes/cleanups that are not
related to the goal of this patchset?

> 
> The main process just waits for events using `epoll()`. When an event is
> received that necessitates a new discovery, a child is forked. This makes it
> possible to fill in the configuration parameters for `do_discover()` without
> interfering with the main process or other discovery tasks running in
> parallel. The program tracks *transport addresses* (called "connections" in
> the code) rather than NVMe controllers. In `--persistent` mode, it tries to
> maintain exactly one persistent discovery connection per transport address.
> 
> Using `epoll()` may look over-engineered at this stage. I hope the better
> flexibility over `poll()` (in particular, the ability to add new event sources
> while waiting) will simplify future extensions and improvements.
> 
> ### Todo
> 
>   * Referrals are not handled perfectly yet. They will be handled by
>     `do_discover()` just as it would when called from **nvme connect-all**, but
>     it would be better to pass referrals back to the main process to make it
>     aware of the additional discovery controller rather than using
>     recursion. The main process would e.g. know if a discovery is already
>     running for the transport address in the referrral.

Agree, this way you can also keep new referrals persistently, which I
don't know if we do today.

>   *  When "add" uevents for nvme controller devices are received, the
>      controller is consistently not in `live` state yet, and attempting to read
>      the `subsysnqn` sysfs attribute returns `(efault)`. While this should
>      arguably be fixed in the kernel, it could be worked around in user space
>      by using timers or polling the `state` sysfs attribute for changes.

This is a bug, what in the code causes this? nothing in controller state
should prevent from this sysfs read from executing correctly...

>   * Parse and handle `discovery.conf` on startup.

This is a must I think, where do you get the known transport addresses
on startup today?

>   * Implement support for RDMA and TCP protocols.

What is needed for supporting them? Not sure I follow (I thought
you mentioned that you tested against linux nvmet-rdma?)

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[Sagi Grimberg]

On 1/26/21 12:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> If autoconnect is enabled, disable the respective udev rules
> by symlinking /run/udev/rules.d to /dev/null, in order to avoid
> the connections being set up by the monitor and the udev workers
> at the same time. This is probably the preferred mode of operation
> for the monitor.
> 
> Users can override this by copying 70-nvmf-autoconnect.rules
> from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
> takes precedence over /run/udev/rules.d).
> 
> If the symlink can't be created for some reason, autoconnect will
> be disabled. There is  only one exception: If
> /run/udev/rules.d/70-nvmf-autoconnect.rules already points to
> /dev/null at startup, autoconnect can be left on, but the symlink
> isn't removed on exit.

Can't we just document that just one should be enabled and skip
the entire thing? Especially when its not critical if both run..

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 16/35] monitor: reset children's signal disposition

[Sagi Grimberg]

No commit message?

> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 30 ++++++++++++++++++++++++++++--
>   1 file changed, 28 insertions(+), 2 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 69c8333..9375a14 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -100,6 +100,8 @@ static void monitor_int_handler(int sig)
>   	must_exit = 1;
>   }
>   
> +static sigset_t orig_sigmask;
> +
>   static int monitor_init_signals(void)
>   {
>   	sigset_t mask;
> @@ -110,7 +112,7 @@ static int monitor_init_signals(void)
>   	 * for events.
>   	 */
>   	sigfillset(&mask);
> -	if (sigprocmask(SIG_BLOCK, &mask, NULL) == -1)
> +	if (sigprocmask(SIG_BLOCK, &mask, &orig_sigmask) == -1)
>   		return -errno;
>   	if (sigaction(SIGTERM, &sa, NULL) == -1)
>   		return -errno;
> @@ -119,6 +121,26 @@ static int monitor_init_signals(void)
>   	return 0;
>   }
>   
> +static int child_reset_signals(void)
> +{
> +	int err = 0;
> +	struct sigaction sa = { .sa_handler = SIG_DFL, };
> +
> +	if (sigaction(SIGTERM, &sa, NULL) == -1)
> +		err = errno;
> +	if (sigaction(SIGINT, &sa, NULL) == -1 && !err)
> +		err = errno;
> +	if (sigaction(SIGCHLD, &sa, NULL) == -1 && !err)
> +		err = errno;
> +
> +	if (sigprocmask(SIG_SETMASK, &orig_sigmask, NULL) == -1 && !err)
> +		err = errno;
> +
> +	if (err)
> +		log(LOG_ERR, "error resetting signal handlers and mask\n");
> +	return -err;
> +}
> +
>   static int monitor_get_fc_uev_props(struct udev_device *ud,
>   				    char *traddr, size_t tra_sz,
>   				    char *host_traddr, size_t htra_sz)
> @@ -176,8 +198,12 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,
>   	if (pid == -1) {
>   		log(LOG_ERR, "failed to fork discovery task: %m");
>   		return -errno;
> -	} else if (pid > 0)
> +	} else if (pid > 0) {
> +		log(LOG_DEBUG, "started discovery task %ld\n", (long)pid);
>   		return 0;
> +	}
> +
> +	child_reset_signals();
>   
>   	log(LOG_NOTICE, "starting %s discovery for %s==>%s(%s)\n",
>   	    transport, host_traddr, traddr, trsvcid ? trsvcid : "none");
> 

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 17/35] monitor: handle SIGCHLD for terminated child processes

[Sagi Grimberg]

No commit message?
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 48 insertions(+), 2 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 9375a14..8db40e6 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -26,6 +26,8 @@
>   #include <limits.h>
>   #include <syslog.h>
>   #include <sys/stat.h>
> +#include <sys/types.h>
> +#include <sys/wait.h>
>   #include <sys/epoll.h>
>   
>   #include "common.h"
> @@ -100,6 +102,13 @@ static void monitor_int_handler(int sig)
>   	must_exit = 1;
>   }
>   
> +static sig_atomic_t got_sigchld;
> +
> +static void monitor_chld_handler(int sig)
> +{
> +	got_sigchld = 1;
> +}
> +
>   static sigset_t orig_sigmask;
>   
>   static int monitor_init_signals(void)
> @@ -118,6 +127,9 @@ static int monitor_init_signals(void)
>   		return -errno;
>   	if (sigaction(SIGINT, &sa, NULL) == -1)
>   		return -errno;
> +	sa.sa_handler = monitor_chld_handler;
> +	if (sigaction(SIGCHLD, &sa, NULL) == -1)
> +		return -errno;
>   	return 0;
>   }
>   
> @@ -336,6 +348,34 @@ static void monitor_handle_uevents(struct udev_monitor *monitor)
>   	}
>   }
>   
> +static void handle_sigchld(void)
> +{
> +	while (true) {
> +		int wstatus;
> +		pid_t pid;
> +
> +		pid = waitpid(-1, &wstatus, WNOHANG);
> +		switch(pid) {
> +		case -1:
> +			if (errno != ECHILD)
> +				log(LOG_ERR, "error in waitpid: %m\n");
> +			return;
> +		case 0:
> +			return;
> +		default:
> +			break;
> +		}
> +		if (!WIFEXITED(wstatus))
> +			log(LOG_WARNING, "child %ld didn't exit normally\n",
> +			    (long)pid);
> +		else if (WEXITSTATUS(wstatus) != 0)
> +			log(LOG_NOTICE, "child %ld exited with status \"%s\"\n",
> +			    (long)pid, strerror(WEXITSTATUS(wstatus)));
> +		else
> +			log(LOG_DEBUG, "child %ld exited normally\n", (long)pid);
> +	};
> +}
> +
>   #define MAX_EVENTS 1
>   static int monitor_main_loop(struct udev_monitor *monitor)
>   {
> @@ -357,13 +397,19 @@ static int monitor_main_loop(struct udev_monitor *monitor)
>   	sigfillset(&ep_mask);
>   	sigdelset(&ep_mask, SIGTERM);
>   	sigdelset(&ep_mask, SIGINT);
> +	sigdelset(&ep_mask, SIGCHLD);
>   	while (1) {
>   		int rc, i;
>   
>   		rc = epoll_pwait(ep_fd, events, MAX_EVENTS, -1, &ep_mask);
>   		if (rc == -1 && errno == EINTR) {
> -			log(LOG_NOTICE, "monitor: exit signal received\n");
> -			return 0;
> +			if (must_exit) {
> +				log(LOG_NOTICE, "monitor: exit signal received\n");
> +				return 0;
> +			} else if (got_sigchld) {
> +				got_sigchld = 0;
> +				handle_sigchld();
> +			}
>   		} else if (rc == -1) {
>   			log(LOG_ERR, "monitor: epoll_wait: %m\n");
>   			return -errno;
> 

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 22/35] conn-db: add simple connection registry

[Sagi Grimberg]

> From: Martin Wilck <mwilck@suse.com>
> 
> The monitor works best if it maintains a discovery controller connection
> to every transport address that provides a discovery subsystem.
> 
> While controllers are easily tracked in sysfs, addresses ("connections"),
> i.e. (transport, traddr, trsvid, host_traddr) tuples, are not. Create
> a simple registry that tracks the state of "connections" and their
> associated discovery controllers.
> 
> A detailed description of the API is provided in the header file conn-db.h.

If this is more flexible, perhaps also the main fabrics 
discover/connect/connect-all can be converted to use it as well.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Martin Wilck]

On Thu, 2021-01-28 at 17:14 -0800, Sagi Grimberg wrote:
> 
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > (Cover letter copied from 
> > https://github.com/linux-nvme/nvme-cli/pull/877)
> > 
> > This patch set adds a new subcommand **nvme monitor**. In this
> > mode,
> > **nvme-cli** runs continuously, monitors events (currently,
> > uevents) relevant
> > for discovery, and optionally autoconnects to newly discovered
> > subsystems.
> > 
> > The monitor mode is suitable to be run in a systemd service. An
> > appropriate
> > unit file is provided. As such, **nvme monitor** can be used as an
> > alternative
> > to the current auto-connection mechanism based on udev rules and
> > systemd
> > template units. If `--autoconnect` is active, **nvme monitor**
> > masks the
> > respective udev rules in order to prevent simultaneous connection
> > attempts
> > from udev and itself.
> 
> I think that a monitor daemon is a good path forward.
> 
> > 
> > This method for discovery and autodetection has some advantages
> > over the
> > current udev-rule based approach:
> > 
> >   * By using the `--persistent` option, users can easily control
> > whether
> >     persistent discovery controllers for discovered transport
> > addresses should
> >     be created and monitored for AEN events. **nvme monitor**
> > watches known
> >     transport addresses, creates discovery controllers as required,
> > and re-uses
> >     existing ones if possible.
> 
> What does that mean?

In general, if the monitor detects a new host_traddr/traddr/trsvcid
tuple, it runs a discovery on it, and keeps the discovery controller
open if --persistent was given. On startup, it scans existing
controllers, and if it finds already existing discovery controllers,
re-uses them. These will not be shut down when the monitor exits.

This allows users fine-grained control about what discovery controllers
to operate persistently. Users who want all discovery controllers to be
persistent just use --persistent. Others can set up those that they
want to have (manually or with a script), and not use --persistent.

The background is that hosts may not need every detected discovery
controller to be persistent. In multipath scenarios, you may see more
discovery subsystems than anything else, and not everyone likes that.
That's a generic issue and unrelated to the monitor, but running the
monitor with --persistent creates discovery controllers that would
otherwise not be visible.

Hope this clarifies it.

> >   * In certain situations, the systemd-based approach may miss
> > events due to
> >     race conditions. This can happen e.g. if an FC remote port is
> > detected, but
> >     shortly after it's detection an FC relogin procedure is
> > necessary e.g. due to
> >     an RSCN. In this case, an `fc_udev_device` uevent will be
> > received on the
> >     first detection and handled by an `nvme connect-all` command
> > run from
> >     `nvmf-connect@.service`. The connection attempt to the rport in
> > question will
> >     fail with "no such device" because of the simultaneous FC
> >     relogin. `nvmf-connect@.service` may not terminate immediately,
> > because it
> >     attempts to establish other connections listed in the Discovery
> > Log page it
> >     retrieved. When the FC relogin eventually finishes, a new
> > uevent will be
> >     received, and `nvmf-connect@` will be started again, but *this
> > has no effect*
> >     if the previous `nvmf-connect@` service hasn't finished yet.
> > This is the
> >     general semantics of systemd services, no easy workaround
> > exists.  **nvme
> >     monitor** doesn't suffer from this problem. If it sees an
> > uevent for a
> >     transport address for which a discovery is already running, it
> > will queue the
> >     handling of this event up and restart the discovery after it's
> > finished.
> 
> While I understand the issue, this reason alone is an overkill for
> doing 
> this.

I agree. But it's not easy to fix the issue otherwise. In the customer
problem where we observed it, I worked around it by adding the udev
seqnum to the "instance name" of the systemd service, thus allowing
several "nvme connect-all" processes to run for the same transport
address simultaneously. But I don't think that would scale well;
the monitor can handle it more cleanly.

> >   * Resource consumption for handling uevents is lower. Instead of
> > running an
> >     udev worker, executing the rules, executing `systemctl start`
> > from the
> >     worker, starting a systemd service, and starting a separate
> > **nvme-cli**
> >     instance, only a single `fork()` operation is necessary. Of
> > course, on the
> >     back side, the monitor itself consumes resources while it's
> > running and
> >     waiting for events. On my system with 8 persistent discovery
> > controllers,
> >     its RSS is ~3MB. CPU consumption is zero as long as no events
> > occur.
> 
> What is the baseline with what we have today?

A meaningful comparsion is difficult and should be done when the
monitor functionality is finalized. I made this statement only to
provide a rough idea of the resource usage, not more.

> >   * **nvme monitor** could be easily extended to handle events for
> > non-FC
> >     transports.
> 
> Which events?

Network discovery, mDNS or the like. I haven't digged into the details
yet.

> > I've tested `fc_udev_device` handling for NVMeoFC with an Ontap
> > target, and
> > AEN handling for RDMA using a Linux **nvmet** target.
> > 
> > ### Implementation notes
> > 
> > I've tried to change the exisiting **nvme-cli** code as little as
> > possible
> > while reusing the code from `fabrics.c`. The majority of changes in
> > the
> > existing code exports formerly static functions and variables, so
> > that they
> > are usable from the monitor code.
> 
> General comment, can you please separate out fixes/cleanups that are
> not
> related to the goal of this patchset?

Which ones are you referring to? 09 and 19? While these are minor
improvements to the existing code, I wouldn't say they qualify as fixes
or cleanups. They aren't necessary without adding the monitor code.

But yes, I can post all changes to existing code separately.

> >   *  When "add" uevents for nvme controller devices are received,
> > the
> >      controller is consistently not in `live` state yet, and
> > attempting to read
> >      the `subsysnqn` sysfs attribute returns `(efault)`. While this
> > should
> >      arguably be fixed in the kernel, it could be worked around in
> > user space
> >      by using timers or polling the `state` sysfs attribute for
> > changes.
> 
> This is a bug, what in the code causes this? nothing in controller
> state
> should prevent from this sysfs read from executing correctly...

I think it can be fixed by making nvme_sysfs_show_subsysnqn() fall back
to ctrl->opts->subsysnqn if ctrl->subsys is NULL.

I'll send a patch. Anyway, it'll take time until this is fixed
everywhere.

> 
> >   * Parse and handle `discovery.conf` on startup.
> 
> This is a must I think, where do you get the known transport
> addresses
> on startup today?

There's a systemd service that runs "nvme connect-all" once during
boot. That exists today. I'm not sure if it should be integrated in the
monitor, perhaps it's good to keep these separate. People who don't
need the monitor can still run the existing service only, whereas for
others, the two would play together just fine.

> 
> >   * Implement support for RDMA and TCP protocols.
> 
> What is needed for supporting them? Not sure I follow (I thought
> you mentioned that you tested against linux nvmet-rdma?)
> 

AENs over existing discovery controllers are supported for all
transports. But there's no support for discovery of new transports
except for NVMeoFC's "fc_udev_device" mechanism.

Regards
Martin





_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[Martin Wilck]

On Thu, 2021-01-28 at 17:52 -0800, Sagi Grimberg wrote:
> 
> 
> On 1/26/21 12:33 PM, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > If autoconnect is enabled, disable the respective udev rules
> > by symlinking /run/udev/rules.d to /dev/null, in order to avoid
> > the connections being set up by the monitor and the udev workers
> > at the same time. This is probably the preferred mode of operation
> > for the monitor.
> > 
> > Users can override this by copying 70-nvmf-autoconnect.rules
> > from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
> > takes precedence over /run/udev/rules.d).
> > 
> > If the symlink can't be created for some reason, autoconnect will
> > be disabled. There is  only one exception: If
> > /run/udev/rules.d/70-nvmf-autoconnect.rules already points to
> > /dev/null at startup, autoconnect can be left on, but the symlink
> > isn't removed on exit.
> 
> Can't we just document that just one should be enabled and skip
> the entire thing? Especially when its not critical if both run..

It's not critical, but it makes no sense to have the rules enabled when
the monitor runs. And having to mask/unmask them manually is a pain. 

It could be done in the systemd service file using ExecStartPre /
ExecStopPost, with small restrictions (we couldn't distinguish whether
the rules were already masked before we started). That would be more
transparent for users.

I'm ok with not doing it automatically if that's generally preferred.

Regards
Martin





_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 22/35] conn-db: add simple connection registry

[Martin Wilck]

On Thu, 2021-01-28 at 17:59 -0800, Sagi Grimberg wrote:
> 
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > The monitor works best if it maintains a discovery controller
> > connection
> > to every transport address that provides a discovery subsystem.
> > 
> > While controllers are easily tracked in sysfs, addresses
> > ("connections"),
> > i.e. (transport, traddr, trsvid, host_traddr) tuples, are not.
> > Create
> > a simple registry that tracks the state of "connections" and their
> > associated discovery controllers.
> > 
> > A detailed description of the API is provided in the header file
> > conn-db.h.
> 
> If this is more flexible, perhaps also the main fabrics 
> discover/connect/connect-all can be converted to use it as well.
> 

Yes. Could we do that in a separate step later.

Regards,
Martin






_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 16/35] monitor: reset children's signal disposition

[Martin Wilck]

On Thu, 2021-01-28 at 17:54 -0800, Sagi Grimberg wrote:
> No commit message?

I'm sorry. I'll add one next time. For #17 likewise.

Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[Sagi Grimberg]

>> On 1/26/21 12:33 PM, mwilck@suse.com wrote:
>>> From: Martin Wilck <mwilck@suse.com>
>>>
>>> If autoconnect is enabled, disable the respective udev rules
>>> by symlinking /run/udev/rules.d to /dev/null, in order to avoid
>>> the connections being set up by the monitor and the udev workers
>>> at the same time. This is probably the preferred mode of operation
>>> for the monitor.
>>>
>>> Users can override this by copying 70-nvmf-autoconnect.rules
>>> from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
>>> takes precedence over /run/udev/rules.d).
>>>
>>> If the symlink can't be created for some reason, autoconnect will
>>> be disabled. There is  only one exception: If
>>> /run/udev/rules.d/70-nvmf-autoconnect.rules already points to
>>> /dev/null at startup, autoconnect can be left on, but the symlink
>>> isn't removed on exit.
>>
>> Can't we just document that just one should be enabled and skip
>> the entire thing? Especially when its not critical if both run..
> 
> It's not critical, but it makes no sense to have the rules enabled when
> the monitor runs. And having to mask/unmask them manually is a pain.
> 
> It could be done in the systemd service file using ExecStartPre /
> ExecStopPost, with small restrictions (we couldn't distinguish whether
> the rules were already masked before we started). That would be more
> transparent for users.

Maybe 'Conflicts' can help here?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 11/35] monitor: add option -A / --autoconnect

[Sagi Grimberg]

> With this option, the monitor will try to connect to newly discovered
> controllers.

Maybe I'm missing something here, but what does it do without
autoconnect?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 18/35] monitor: add "--persistent/-p" flag

[Sagi Grimberg]

> Without this flag, AENs on created controllers won't be received.
> While --persistent is probably the behavior most users will want,
> it has to be explicitly activated, to be consistent with the
> connect-all command.

What is the usage model without this flag?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 32/35] nvmf-autoconnect: add unit file for nvme-monitor.service

[Sagi Grimberg]

> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   nvmf-autoconnect/systemd/nvme-monitor.service | 17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
>   create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
> 
> diff --git a/nvmf-autoconnect/systemd/nvme-monitor.service b/nvmf-autoconnect/systemd/nvme-monitor.service
> new file mode 100644
> index 0000000..d85b133
> --- /dev/null
> +++ b/nvmf-autoconnect/systemd/nvme-monitor.service
> @@ -0,0 +1,17 @@
> +[Unit]
> +Description=NVMe Event Monitor for Automatical Subsystem Connection
> +Documentation=man:nvme-monitor(1)
> +DefaultDependencies=false
> +Requires=systemd-udevd-kernel.socket
> +RequiresMountsFor=/sys
> +Conflicts=shutdown.target
> +After=systemd-udevd-kernel.socket
> +Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-connections.service
> +
> +[Service]
> +Type=simple
> +ExecStart=/usr/sbin/nvme monitor -A -p

I've seen issues with assuming nvme sits in /usr/sbin/

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 33/35] nvme-connect-all(1): fix documentation for --quiet/-S

[Sagi Grimberg]

Cleanup unrelated patch

On 1/26/21 12:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   Documentation/nvme-connect-all.1    |  8 ++++----
>   Documentation/nvme-connect-all.html | 10 +++++-----
>   Documentation/nvme-connect-all.txt  |  4 ++--
>   3 files changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/nvme-connect-all.1 b/Documentation/nvme-connect-all.1
> index 91d6896..a53a017 100644
> --- a/Documentation/nvme-connect-all.1
> +++ b/Documentation/nvme-connect-all.1
> @@ -2,12 +2,12 @@
>   .\"     Title: nvme-connect-all
>   .\"    Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author]
>   .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/>
> -.\"      Date: 10/20/2020
> +.\"      Date: 01/20/2021
>   .\"    Manual: NVMe Manual
>   .\"    Source: NVMe
>   .\"  Language: English
>   .\"
> -.TH "NVME\-CONNECT\-ALL" "1" "10/20/2020" "NVMe" "NVMe Manual"
> +.TH "NVME\-CONNECT\-ALL" "1" "01/20/2021" "NVMe" "NVMe Manual"
>   .\" -----------------------------------------------------------------
>   .\" * Define some portability stuff
>   .\" -----------------------------------------------------------------
> @@ -51,7 +51,7 @@ nvme-connect-all \- Discover and Connect to Fabrics controllers\&.
>                   [\-\-queue\-size=<#>         | \-Q <#>]
>                   [\-\-matching               | \-m]
>                   [\-\-persistent             | \-p]
> -                [\-\-quiet                  | \-q]
> +                [\-\-quiet                  | \-S]
>   .fi
>   .SH "DESCRIPTION"
>   .sp
> @@ -193,7 +193,7 @@ If a traddr was specified on the command line or in the configuration file, only
>   Don\(cqt remove the discovery controller after retrieving the discovery log page\&.
>   .RE
>   .PP
> -\-q, \-\-quiet
> +\-S, \-\-quiet
>   .RS 4
>   Suppress error messages\&.
>   .RE
> diff --git a/Documentation/nvme-connect-all.html b/Documentation/nvme-connect-all.html
> index beae013..f779c7b 100644
> --- a/Documentation/nvme-connect-all.html
> +++ b/Documentation/nvme-connect-all.html
> @@ -4,7 +4,7 @@
>   <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
>   <head>
>   <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
> -<meta name="generator" content="AsciiDoc 8.6.10" />
> +<meta name="generator" content="AsciiDoc" />
>   <title>nvme-connect-all(1)</title>
>   <style type="text/css">
>   /* Shared CSS for AsciiDoc xhtml11 and html5 backends */
> @@ -436,7 +436,7 @@ thead, p.table.header {
>   p.table {
>     margin-top: 0;
>   }
> -/* Because the table frame attribute is overriden by CSS in most browsers. */
> +/* Because the table frame attribute is overridden by CSS in most browsers. */
>   div.tableblock > table[frame="void"] {
>     border-style: none;
>   }
> @@ -768,7 +768,7 @@ nvme-connect-all(1) Manual Page
>                   [--queue-size=&lt;#&gt;         | -Q &lt;#&gt;]
>                   [--matching               | -m]
>                   [--persistent             | -p]
> -                [--quiet                  | -q]</pre>
> +                [--quiet                  | -S]</pre>
>   <div class="attribution">
>   </div></div>
>   </div>
> @@ -1048,7 +1048,7 @@ cellspacing="0" cellpadding="4">
>   </p>
>   </dd>
>   <dt class="hdlist1">
> --q
> +-S
>   </dt>
>   <dt class="hdlist1">
>   --quiet
> @@ -1115,7 +1115,7 @@ nvme-connect(1)</p></div>
>   <div id="footer">
>   <div id="footer-text">
>   Last updated
> - 2020-10-20 16:47:21 PDT
> + 2021-01-20 23:40:57 CET
>   </div>
>   </div>
>   </body>
> diff --git a/Documentation/nvme-connect-all.txt b/Documentation/nvme-connect-all.txt
> index 128f336..820dd6c 100644
> --- a/Documentation/nvme-connect-all.txt
> +++ b/Documentation/nvme-connect-all.txt
> @@ -27,7 +27,7 @@ SYNOPSIS
>   		[--queue-size=<#>         | -Q <#>]
>   		[--matching               | -m]
>   		[--persistent             | -p]
> -		[--quiet                  | -q]
> +		[--quiet                  | -S]
>   
>   DESCRIPTION
>   -----------
> @@ -154,7 +154,7 @@ OPTIONS
>   	Don't remove the discovery controller after retrieving the discovery
>   	log page.
>   
> --q::
> +-S::
>   --quiet::
>   	Suppress error messages.
>   
> 

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

> From: Martin Wilck <mwilck@suse.com>
> 
> With this option, "nvme monitor" will keep created discovery controllers
> running even after it exits. If running in the initrd, this is will be
> done automatically. That has the advantage that the new monitor instance
> that will be started automatically after switching root can simply take
> over the already created discovery controllers.

Why not make it the default?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 11/35] monitor: add option -A / --autoconnect

[Martin Wilck]

On Fri, 2021-01-29 at 10:59 -0800, Sagi Grimberg wrote:
> 
> > With this option, the monitor will try to connect to newly
> > discovered
> > controllers.
> 
> Maybe I'm missing something here, but what does it do without
> autoconnect?
> 

Basically it just runs "nvme discover". When events come in,
it runs the discovery and prints the results, like "discover" would.

We can change that of course.

Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 18/35] monitor: add "--persistent/-p" flag

[Martin Wilck]

On Fri, 2021-01-29 at 11:02 -0800, Sagi Grimberg wrote:
> 
> > Without this flag, AENs on created controllers won't be received.
> > While --persistent is probably the behavior most users will want,
> > it has to be explicitly activated, to be consistent with the
> > connect-all command.
> 
> What is the usage model without this flag?
> 

 - listen to other events that would require a discovery, currently
simply fc_udev_device. In the future, network events could be added.
 - listen for AENs on persistent discovery controllers that are created
by other means, e.g. manually.
   
See also our discussion about the cover letter of this series.

Martin




_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 32/35] nvmf-autoconnect: add unit file for nvme-monitor.service

[Martin Wilck]

On Fri, 2021-01-29 at 11:08 -0800, Sagi Grimberg wrote:
> 
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > ---
> >   nvmf-autoconnect/systemd/nvme-monitor.service | 17
> > +++++++++++++++++
> >   1 file changed, 17 insertions(+)
> >   create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
> > 
> > diff --git a/nvmf-autoconnect/systemd/nvme-monitor.service b/nvmf-
> > autoconnect/systemd/nvme-monitor.service
> > new file mode 100644
> > index 0000000..d85b133
> > --- /dev/null
> > +++ b/nvmf-autoconnect/systemd/nvme-monitor.service
> > @@ -0,0 +1,17 @@
> > +[Unit]
> > +Description=NVMe Event Monitor for Automatical Subsystem
> > Connection
> > +Documentation=man:nvme-monitor(1)
> > +DefaultDependencies=false
> > +Requires=systemd-udevd-kernel.socket
> > +RequiresMountsFor=/sys
> > +Conflicts=shutdown.target
> > +After=systemd-udevd-kernel.socket
> > +Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-
> > connections.service
> > +
> > +[Service]
> > +Type=simple
> > +ExecStart=/usr/sbin/nvme monitor -A -p
> 
> I've seen issues with assuming nvme sits in /usr/sbin/

Sure, this can be fixed. More fine tuning is possible, too. I learned
meanwhile that the dependency on systemd-udevd-kernel.socket is
unnecessary.

Regards
Martin




_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Fri, 2021-01-29 at 11:10 -0800, Sagi Grimberg wrote:
> 
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > With this option, "nvme monitor" will keep created discovery
> > controllers
> > running even after it exits. If running in the initrd, this is will
> > be
> > done automatically. That has the advantage that the new monitor
> > instance
> > that will be started automatically after switching root can simply
> > take
> > over the already created discovery controllers.
> 
> Why not make it the default?

Well, we've had a lot of discussions about persistent discovery
connections not being universally loved. So I thought I'd make cleaning
up the default.

All this can be discussed of course, that's why I called it an RFC.

Martin





_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Sagi Grimberg]

>>>
>>> This method for discovery and autodetection has some advantages
>>> over the
>>> current udev-rule based approach:
>>>
>>>    * By using the `--persistent` option, users can easily control
>>> whether
>>>      persistent discovery controllers for discovered transport
>>> addresses should
>>>      be created and monitored for AEN events. **nvme monitor**
>>> watches known
>>>      transport addresses, creates discovery controllers as required,
>>> and re-uses
>>>      existing ones if possible.
>>
>> What does that mean?
> 
> In general, if the monitor detects a new host_traddr/traddr/trsvcid
> tuple, it runs a discovery on it, and keeps the discovery controller
> open if --persistent was given. On startup, it scans existing
> controllers, and if it finds already existing discovery controllers,
> re-uses them. These will not be shut down when the monitor exits.

And if it doesn't run with --persistent it deletes them? even if these
weren't created by it?

And, if none exist where does it get new discovery controller details?
Today we use discovery.conf for that.

> This allows users fine-grained control about what discovery controllers
> to operate persistently. Users who want all discovery controllers to be
> persistent just use --persistent. Others can set up those that they
> want to have (manually or with a script), and not use --persistent.
> 
> The background is that hosts may not need every detected discovery
> controller to be persistent. In multipath scenarios, you may see more
> discovery subsystems than anything else, and not everyone likes that.
> That's a generic issue and unrelated to the monitor, but running the
> monitor with --persistent creates discovery controllers that would
> otherwise not be visible.
> 
> Hope this clarifies it.

Well, How do people expect to know if stuff change without a persistent
discovery controller? Not sure if people may see more discovery
controllers this is a real issue given what they are getting from it.

But OK.

>>>    * In certain situations, the systemd-based approach may miss
>>> events due to
>>>      race conditions. This can happen e.g. if an FC remote port is
>>> detected, but
>>>      shortly after it's detection an FC relogin procedure is
>>> necessary e.g. due to
>>>      an RSCN. In this case, an `fc_udev_device` uevent will be
>>> received on the
>>>      first detection and handled by an `nvme connect-all` command
>>> run from
>>>      `nvmf-connect@.service`. The connection attempt to the rport in
>>> question will
>>>      fail with "no such device" because of the simultaneous FC
>>>      relogin. `nvmf-connect@.service` may not terminate immediately,
>>> because it
>>>      attempts to establish other connections listed in the Discovery
>>> Log page it
>>>      retrieved. When the FC relogin eventually finishes, a new
>>> uevent will be
>>>      received, and `nvmf-connect@` will be started again, but *this
>>> has no effect*
>>>      if the previous `nvmf-connect@` service hasn't finished yet.
>>> This is the
>>>      general semantics of systemd services, no easy workaround
>>> exists.  **nvme
>>>      monitor** doesn't suffer from this problem. If it sees an
>>> uevent for a
>>>      transport address for which a discovery is already running, it
>>> will queue the
>>>      handling of this event up and restart the discovery after it's
>>> finished.
>>
>> While I understand the issue, this reason alone is an overkill for
>> doing
>> this.
> 
> I agree. But it's not easy to fix the issue otherwise. In the customer
> problem where we observed it, I worked around it by adding the udev
> seqnum to the "instance name" of the systemd service, thus allowing
> several "nvme connect-all" processes to run for the same transport
> address simultaneously. But I don't think that would scale well;
> the monitor can handle it more cleanly.

Still changing the entire thing for a corner case...

> 
>>>    * Resource consumption for handling uevents is lower. Instead of
>>> running an
>>>      udev worker, executing the rules, executing `systemctl start`
>>> from the
>>>      worker, starting a systemd service, and starting a separate
>>> **nvme-cli**
>>>      instance, only a single `fork()` operation is necessary. Of
>>> course, on the
>>>      back side, the monitor itself consumes resources while it's
>>> running and
>>>      waiting for events. On my system with 8 persistent discovery
>>> controllers,
>>>      its RSS is ~3MB. CPU consumption is zero as long as no events
>>> occur.
>>
>> What is the baseline with what we have today?
> 
> A meaningful comparsion is difficult and should be done when the
> monitor functionality is finalized. I made this statement only to
> provide a rough idea of the resource usage, not more.

You mention that the utilization is lower than what we do today, hence
my question is by how much?

>>>    * **nvme monitor** could be easily extended to handle events for
>>> non-FC
>>>      transports.
>>
>> Which events?
> 
> Network discovery, mDNS or the like. I haven't digged into the details
> yet.

Yes, that is possible, would probably be easier to do this in a higher
level language but libavahi can also do this...

>>> I've tested `fc_udev_device` handling for NVMeoFC with an Ontap
>>> target, and
>>> AEN handling for RDMA using a Linux **nvmet** target.
>>>
>>> ### Implementation notes
>>>
>>> I've tried to change the exisiting **nvme-cli** code as little as
>>> possible
>>> while reusing the code from `fabrics.c`. The majority of changes in
>>> the
>>> existing code exports formerly static functions and variables, so
>>> that they
>>> are usable from the monitor code.
>>
>> General comment, can you please separate out fixes/cleanups that are
>> not
>> related to the goal of this patchset?
> 
> Which ones are you referring to? 09 and 19? While these are minor
> improvements to the existing code, I wouldn't say they qualify as fixes
> or cleanups. They aren't necessary without adding the monitor code.

Patches that are not directly related to the goal of this should
be splitted out to ease the review.

> But yes, I can post all changes to existing code separately.
> 
>>>    *  When "add" uevents for nvme controller devices are received,
>>> the
>>>       controller is consistently not in `live` state yet, and
>>> attempting to read
>>>       the `subsysnqn` sysfs attribute returns `(efault)`. While this
>>> should
>>>       arguably be fixed in the kernel, it could be worked around in
>>> user space
>>>       by using timers or polling the `state` sysfs attribute for
>>> changes.
>>
>> This is a bug, what in the code causes this? nothing in controller
>> state
>> should prevent from this sysfs read from executing correctly...
> 
> I think it can be fixed by making nvme_sysfs_show_subsysnqn() fall back
> to ctrl->opts->subsysnqn if ctrl->subsys is NULL.
> 
> I'll send a patch. Anyway, it'll take time until this is fixed
> everywhere.

Thanks.

> 
>>
>>>    * Parse and handle `discovery.conf` on startup.
>>
>> This is a must I think, where do you get the known transport
>> addresses
>> on startup today?
> 
> There's a systemd service that runs "nvme connect-all" once during
> boot. That exists today. I'm not sure if it should be integrated in the
> monitor, perhaps it's good to keep these separate. People who don't
> need the monitor can still run the existing service only, whereas for
> others, the two would play together just fine.

Then doesn't this service need to run after it?

>>>    * Implement support for RDMA and TCP protocols.
>>
>> What is needed for supporting them? Not sure I follow (I thought
>> you mentioned that you tested against linux nvmet-rdma?)
>>
> 
> AENs over existing discovery controllers are supported for all
> transports. But there's no support for discovery of new transports
> except for NVMeoFC's "fc_udev_device" mechanism.

I see. thanks.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 11/35] monitor: add option -A / --autoconnect

[Sagi Grimberg]

>>> With this option, the monitor will try to connect to newly
>>> discovered
>>> controllers.
>>
>> Maybe I'm missing something here, but what does it do without
>> autoconnect?
>>
> 
> Basically it just runs "nvme discover". When events come in,
> it runs the discovery and prints the results, like "discover" would.
> 
> We can change that of course.

But its a daemon, seems not very useful, but ok..

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

>>> With this option, "nvme monitor" will keep created discovery
>>> controllers
>>> running even after it exits. If running in the initrd, this is will
>>> be
>>> done automatically. That has the advantage that the new monitor
>>> instance
>>> that will be started automatically after switching root can simply
>>> take
>>> over the already created discovery controllers.
>>
>> Why not make it the default?
> 
> Well, we've had a lot of discussions about persistent discovery
> connections not being universally loved. So I thought I'd make cleaning
> up the default.

Heh... well, I didn't see thus far of any other good suggestion on how
to propagate discovery change log events reliably (I've seen some bad
ones). And we are assuming that hosts need to know about these
changes...

The default matters here. The spec tells us how to learn about
change log events so I would suggest we follow it.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Martin Wilck]

On Fri, 2021-01-29 at 12:08 -0800, Sagi Grimberg wrote:
> 
> > > > 
> > > > This method for discovery and autodetection has some advantages
> > > > over the
> > > > current udev-rule based approach:
> > > > 
> > > >    * By using the `--persistent` option, users can easily
> > > > control
> > > > whether
> > > >      persistent discovery controllers for discovered transport
> > > > addresses should
> > > >      be created and monitored for AEN events. **nvme monitor**
> > > > watches known
> > > >      transport addresses, creates discovery controllers as
> > > > required,
> > > > and re-uses
> > > >      existing ones if possible.
> > > 
> > > What does that mean?
> > 
> > In general, if the monitor detects a new host_traddr/traddr/trsvcid
> > tuple, it runs a discovery on it, and keeps the discovery
> > controller
> > open if --persistent was given. On startup, it scans existing
> > controllers, and if it finds already existing discovery
> > controllers,
> > re-uses them. These will not be shut down when the monitor exits.
> 
> And if it doesn't run with --persistent it deletes them? even if
> these
> weren't created by it?

No, not those. That's what I wanted to express with "These will not be
shut down when the monitor exits."

It's not trivial to determine reliably which controller the monitor
created and which it didn't. The current code assumes that all
discovery controllers that didn't exist at startup were created by the
monitor. We can improve the intelligence of the tool in that area, of
course. Currently it makes this dumb assumption.

> 
> And, if none exist where does it get new discovery controller
> details?
> Today we use discovery.conf for that.

As I said, this can be done with a separate "nvme connect-all" call,
which could be run from an ExecStartPre or from a separate service.
Or we can integrate the functionality in the monitor. I don't have a
strong opinion either way. I can adapt to what you guys prefer.

> 
> > This allows users fine-grained control about what discovery
> > controllers
> > to operate persistently. Users who want all discovery controllers
> > to be
> > persistent just use --persistent. Others can set up those that they
> > want to have (manually or with a script), and not use --persistent.
> > 
> > The background is that hosts may not need every detected discovery
> > controller to be persistent. In multipath scenarios, you may see
> > more
> > discovery subsystems than anything else, and not everyone likes
> > that.
> > That's a generic issue and unrelated to the monitor, but running
> > the
> > monitor with --persistent creates discovery controllers that would
> > otherwise not be visible.
> > 
> > Hope this clarifies it.
> 
> Well, How do people expect to know if stuff change without a
> persistent
> discovery controller? Not sure if people may see more discovery
> controllers this is a real issue given what they are getting from it.

You're right. At this very early stage, I wanted to give users the
freedom of choice, and not force dozens of discovery controller
connections upon everyone. If there's consensus that the discovery
connections should always be created, fine with me. 


> > I agree. But it's not easy to fix the issue otherwise. In the
> > customer
> > problem where we observed it, I worked around it by adding the udev
> > seqnum to the "instance name" of the systemd service, thus allowing
> > several "nvme connect-all" processes to run for the same transport
> > address simultaneously. But I don't think that would scale well;
> > the monitor can handle it more cleanly.
> 
> Still changing the entire thing for a corner case...

It's just one puzzle piece.

> > > >    * Resource consumption for handling uevents is lower.
> > > > Instead of
> > > > running an
> > > >      udev worker, executing the rules, executing `systemctl
> > > > start`
> > > > from the
> > > >      worker, starting a systemd service, and starting a
> > > > separate
> > > > **nvme-cli**
> > > >      instance, only a single `fork()` operation is necessary.
> > > > Of
> > > > course, on the
> > > >      back side, the monitor itself consumes resources while
> > > > it's
> > > > running and
> > > >      waiting for events. On my system with 8 persistent
> > > > discovery
> > > > controllers,
> > > >      its RSS is ~3MB. CPU consumption is zero as long as no
> > > > events
> > > > occur.
> > > 
> > > What is the baseline with what we have today?
> > 
> > A meaningful comparsion is difficult and should be done when the
> > monitor functionality is finalized. I made this statement only to
> > provide a rough idea of the resource usage, not more.
> 
> You mention that the utilization is lower than what we do today,
> hence
> my question is by how much?

I'll try to figure it out and come up with something. Sorry for the
hand-waving assertion. I admint It was speculative for the most part.

> > > >    * **nvme monitor** could be easily extended to handle events
> > > > for
> > > > non-FC
> > > >      transports.
> > > 
> > > Which events?
> > 
> > Network discovery, mDNS or the like. I haven't digged into the
> > details
> > yet.
> 
> Yes, that is possible, would probably be easier to do this in a
> higher
> level language but libavahi can also do this...

Enzo Matsumiya has been working on it, and we've been discussing how to
integrate his functionality into the monitor.

> > > 
> > > >    * Parse and handle `discovery.conf` on startup.
> > > 
> > > This is a must I think, where do you get the known transport
> > > addresses
> > > on startup today?
> > 
> > There's a systemd service that runs "nvme connect-all" once during
> > boot. That exists today. I'm not sure if it should be integrated in
> > the
> > monitor, perhaps it's good to keep these separate. People who don't
> > need the monitor can still run the existing service only, whereas
> > for
> > others, the two would play together just fine.
> 
> Then doesn't this service need to run after it?

Not necessarily. It can handle controllers that are created by other
processes while it's running.

Thanks for your comments,
Martin




_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Fri, 2021-01-29 at 12:16 -0800, Sagi Grimberg wrote:
> 
> > > > With this option, "nvme monitor" will keep created discovery
> > > > controllers
> > > > running even after it exits. If running in the initrd, this is
> > > > will
> > > > be
> > > > done automatically. That has the advantage that the new monitor
> > > > instance
> > > > that will be started automatically after switching root can
> > > > simply
> > > > take
> > > > over the already created discovery controllers.
> > > 
> > > Why not make it the default?
> > 
> > Well, we've had a lot of discussions about persistent discovery
> > connections not being universally loved. So I thought I'd make
> > cleaning
> > up the default.
> 
> Heh... well, I didn't see thus far of any other good suggestion on
> how
> to propagate discovery change log events reliably (I've seen some bad
> ones). And we are assuming that hosts need to know about these
> changes...
> 
> The default matters here. The spec tells us how to learn about
> change log events so I would suggest we follow it.
> 

No problem, I can change this.

But I'm not quite following you here, because this option is about
keeping controllers up after the monitor exits. Given that the monitor
would be the instance that receives the AENs and acts upon them, what's
the point of keeping them alive after it has terminated?

Martin





_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

>>>>> With this option, "nvme monitor" will keep created discovery
>>>>> controllers
>>>>> running even after it exits. If running in the initrd, this is
>>>>> will
>>>>> be
>>>>> done automatically. That has the advantage that the new monitor
>>>>> instance
>>>>> that will be started automatically after switching root can
>>>>> simply
>>>>> take
>>>>> over the already created discovery controllers.
>>>>
>>>> Why not make it the default?
>>>
>>> Well, we've had a lot of discussions about persistent discovery
>>> connections not being universally loved. So I thought I'd make
>>> cleaning
>>> up the default.
>>
>> Heh... well, I didn't see thus far of any other good suggestion on
>> how
>> to propagate discovery change log events reliably (I've seen some bad
>> ones). And we are assuming that hosts need to know about these
>> changes...
>>
>> The default matters here. The spec tells us how to learn about
>> change log events so I would suggest we follow it.
>>
> 
> No problem, I can change this.
> 
> But I'm not quite following you here, because this option is about
> keeping controllers up after the monitor exits. Given that the monitor
> would be the instance that receives the AENs and acts upon them, what's
> the point of keeping them alive after it has terminated?

It may run again at some point.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Fri, 2021-01-29 at 12:45 -0800, Sagi Grimberg wrote:
> 
> > > > > 
> > 
> > But I'm not quite following you here, because this option is about
> > keeping controllers up after the monitor exits. Given that the
> > monitor
> > would be the instance that receives the AENs and acts upon them,
> > what's
> > the point of keeping them alive after it has terminated?
> 
> It may run again at some point.
> 

If it does, it will recreate discovery controllers for every
host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only
necessary for addresses on which no regular (non-discovery) connection
exists.

Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

>>> But I'm not quite following you here, because this option is about
>>> keeping controllers up after the monitor exits. Given that the
>>> monitor
>>> would be the instance that receives the AENs and acts upon them,
>>> what's
>>> the point of keeping them alive after it has terminated?
>>
>> It may run again at some point.
>>
> 
> If it does, it will recreate discovery controllers for every
> host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only
> necessary for addresses on which no regular (non-discovery) connection
> exists.

Wait, Maybe I'm missing something here, but are you saying that for
every traddr/trsvcid it finds (both nvm and discovery) it will attempt
to connect a discovery controller?

If so, this is absolutely wrong.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Fri, 2021-01-29 at 12:57 -0800, Sagi Grimberg wrote:
> 
> > > > But I'm not quite following you here, because this option is
> > > > about
> > > > keeping controllers up after the monitor exits. Given that the
> > > > monitor
> > > > would be the instance that receives the AENs and acts upon
> > > > them,
> > > > what's
> > > > the point of keeping them alive after it has terminated?
> > > 
> > > It may run again at some point.
> > > 
> > 
> > If it does, it will recreate discovery controllers for every
> > host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are
> > only
> > necessary for addresses on which no regular (non-discovery)
> > connection
> > exists.
> 
> Wait, Maybe I'm missing something here, but are you saying that for
> every traddr/trsvcid it finds (both nvm and discovery) it will
> attempt
> to connect a discovery controller?
> 
> If so, this is absolutely wrong.

Currently, it tries to do that on startup, if (and only if) the 
--startup option is given. My expectation was that the connection
attempts would simply fail if there was no discovery subsystem to
connect to. Anyway, it's not the default behavior, and can be dropped
completely if it's so bad that we shouldn't ever attempt to do it.

If the service is started early on during boot, and event-based
discovery works (i.e. we also have the mDNS part in place), this won't
be necessary of course.

Regards
Martin






_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 26/35] monitor: implement starting discovery controllers on startup

[Sagi Grimberg]

> If the -U/--startup option is used, nvme monitor looks for exsiting
> nvme controllers on startup, and runs a discovery on the associated
> transport address (connection); i.e. tries to connect to a discovery
> subsystem on the same address, retrieve the discovery log pages,
> and (if --autoconnect is given) connect all listed controllers.

Oh no no no no... This is a completely wrong assumption to make. Nothing
suggest that an nvme controller will have a discovery controller on the
same port.

There is a discovery controller and an NVM controller, the only
connection between the two is that a discovery controller can
refer to an nvme controller (or a different discovery controller).

Information about discovery controllers may be obtained in different
ways, but it absolutely cannot be obtained by any nvme controller.

Now I understand a bit more on the --keep discussion. The monitor
should today get discovery controller endpoints from a conf file
(maybe discovery.conf) and in the future by any other automatic
way, not even looking into the currently connected nvme controllers.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

>>> If it does, it will recreate discovery controllers for every
>>> host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are
>>> only
>>> necessary for addresses on which no regular (non-discovery)
>>> connection
>>> exists.
>>
>> Wait, Maybe I'm missing something here, but are you saying that for
>> every traddr/trsvcid it finds (both nvm and discovery) it will
>> attempt
>> to connect a discovery controller?
>>
>> If so, this is absolutely wrong.
> 
> Currently, it tries to do that on startup, if (and only if) the
> --startup option is given. My expectation was that the connection
> attempts would simply fail if there was no discovery subsystem to
> connect to. Anyway, it's not the default behavior, and can be dropped
> completely if it's so bad that we shouldn't ever attempt to do it.

IMO it needs to be dropped. I didn't understand this at first because
it never even occurred to me that such an assumption can be even made.

> If the service is started early on during boot, and event-based
> discovery works (i.e. we also have the mDNS part in place), this won't
> be necessary of course.

This isn't necessary regardless. At best the discovery controller
endpoints should be obtained from discovery.conf or equivalent.

I don't even see how does this help in early boot anyways, how
do the existing controllers get connected?

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 26/35] monitor: implement starting discovery controllers on startup

[Martin Wilck]

On Fri, 2021-01-29 at 13:06 -0800, Sagi Grimberg wrote:
> 
> > If the -U/--startup option is used, nvme monitor looks for exsiting
> > nvme controllers on startup, and runs a discovery on the associated
> > transport address (connection); i.e. tries to connect to a
> > discovery
> > subsystem on the same address, retrieve the discovery log pages,
> > and (if --autoconnect is given) connect all listed controllers.
> 
> Oh no no no no... This is a completely wrong assumption to make.
> Nothing
> suggest that an nvme controller will have a discovery controller on
> the
> same port.
> 
> There is a discovery controller and an NVM controller, the only
> connection between the two is that a discovery controller can
> refer to an nvme controller (or a different discovery controller).
> 
> Information about discovery controllers may be obtained in different
> ways, but it absolutely cannot be obtained by any nvme controller.
> 
> Now I understand a bit more on the --keep discussion. The monitor
> should today get discovery controller endpoints from a conf file
> (maybe discovery.conf) and in the future by any other automatic
> way, not even looking into the currently connected nvme controllers.
> 

I hear you. I'll drop this in the next version, no problem. 

I didn't expect that probing for a discovery controller was a problem.
IIUC, if we get an fc_udev_device event for a NVMe remote port, we
can't be sure that that remote port offers a discovery controller,
either. We try to connect, and give up if it fails.

Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Fri, 2021-01-29 at 13:11 -0800, Sagi Grimberg wrote:
> 
> > > > If it does, it will recreate discovery controllers for every
> > > > host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics
> > > > are
> > > > only
> > > > necessary for addresses on which no regular (non-discovery)
> > > > connection
> > > > exists.
> > > 
> > > Wait, Maybe I'm missing something here, but are you saying that
> > > for
> > > every traddr/trsvcid it finds (both nvm and discovery) it will
> > > attempt
> > > to connect a discovery controller?
> > > 
> > > If so, this is absolutely wrong.
> > 
> > Currently, it tries to do that on startup, if (and only if) the
> > --startup option is given. My expectation was that the connection
> > attempts would simply fail if there was no discovery subsystem to
> > connect to. Anyway, it's not the default behavior, and can be
> > dropped
> > completely if it's so bad that we shouldn't ever attempt to do it.
> 
> IMO it needs to be dropped. I didn't understand this at first because
> it never even occurred to me that such an assumption can be even
> made.
> 
> > If the service is started early on during boot, and event-based
> > discovery works (i.e. we also have the mDNS part in place), this
> > won't
> > be necessary of course.
> 
> This isn't necessary regardless. At best the discovery controller
> endpoints should be obtained from discovery.conf or equivalent.

Understood.

> I don't even see how does this help in early boot anyways, how
> do the existing controllers get connected?

For FC, it works today. You can start the monitor in the initrd, it
will receive the fc_udev_device events, and autoconnect (just like
traditional udev/systemd based discovery). For other transports, it'd
be more work, as we'd need to do mDNS discovery from an initrd
environment. I wouldn't say it's impossible though.

Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 26/35] monitor: implement starting discovery controllers on startup

[Sagi Grimberg]

>>> If the -U/--startup option is used, nvme monitor looks for exsiting
>>> nvme controllers on startup, and runs a discovery on the associated
>>> transport address (connection); i.e. tries to connect to a
>>> discovery
>>> subsystem on the same address, retrieve the discovery log pages,
>>> and (if --autoconnect is given) connect all listed controllers.
>>
>> Oh no no no no... This is a completely wrong assumption to make.
>> Nothing
>> suggest that an nvme controller will have a discovery controller on
>> the
>> same port.
>>
>> There is a discovery controller and an NVM controller, the only
>> connection between the two is that a discovery controller can
>> refer to an nvme controller (or a different discovery controller).
>>
>> Information about discovery controllers may be obtained in different
>> ways, but it absolutely cannot be obtained by any nvme controller.
>>
>> Now I understand a bit more on the --keep discussion. The monitor
>> should today get discovery controller endpoints from a conf file
>> (maybe discovery.conf) and in the future by any other automatic
>> way, not even looking into the currently connected nvme controllers.
>>
> 
> I hear you. I'll drop this in the next version, no problem.
> 
> I didn't expect that probing for a discovery controller was a problem.

It's not a "problem" in the sense that nothing devastating will happen.
it's just wrong. One can also randomize a transport endpoint and attempt
to connect to it.

> IIUC, if we get an fc_udev_device event for a NVMe remote port, we
> can't be sure that that remote port offers a discovery controller,
> either. We try to connect, and give up if it fails.

I'm assuming that this is what the spec mandates the host should do,
which is not the case here.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Sagi Grimberg]

>> I don't even see how does this help in early boot anyways, how
>> do the existing controllers get connected?
> 
> For FC, it works today. You can start the monitor in the initrd, it
> will receive the fc_udev_device events, and autoconnect (just like
> traditional udev/systemd based discovery). For other transports, it'd
> be more work, as we'd need to do mDNS discovery from an initrd
> environment. I wouldn't say it's impossible though.

I was referring to the nvme controller walk and connect a discovery
controller to that endpoint. I thought you meant that this helps
you at early boot time.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 05/35] monitor: add uevent filters

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/monitor.c b/monitor.c
> index 63e26bc..a6c8905 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -58,6 +58,10 @@ static int create_udev_monitor(struct udev_monitor **pmon)
>   	if (!mon)
>   		return errno ? -errno : -ENOMEM;
>   
> +	/* Add match for NVMe controller devices */
> +	ret = udev_monitor_filter_add_match_subsystem_devtype(mon, "nvme", NULL);
> +	/* Add match for fc_udev_device */
> +	ret = udev_monitor_filter_add_match_subsystem_devtype(mon, "fc", NULL);
>   	/*
>   	 * This fails in unpriviliged mode. Use the same value as udevd.
>   	 * We may able to decrease this buffer size later.
> 
I guess you could merge all these five patches together; no need to keep 
them separate.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/35] monitor: Create a log() macro.

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> As this is a long running program, we need to make the log output
> configurable. First step: replace fprintf() by log(). The log level
> and printing of time stamps can be configured at run time using
> global variables. These will live in fabrics.c.
> 
> Allow toggling function name printing at build time.
> Printing the function name is useful for development, but perhaps
> not desired for production.
> Put '#define LOG_FUNCNAME' before '#include "log.h"' to switch it on.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   log.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 44 insertions(+)
>   create mode 100644 log.h
> 
> diff --git a/log.h b/log.h
> new file mode 100644
> index 0000000..2017731
> --- /dev/null
> +++ b/log.h
> @@ -0,0 +1,44 @@
> +#ifndef _LOG_H
> +#define _LOG_H
> +
> +#ifndef MAX_LOGLEVEL
> +#  define MAX_LOGLEVEL LOG_DEBUG
> +#endif
> +#ifndef DEFAULT_LOGLEVEL
> +#  define DEFAULT_LOGLEVEL LOG_NOTICE
> +#endif
> +
> +#ifdef LOG_FUNCNAME
> +#define _func_fmt "%s: "
> +#define _func_arg __func__
> +#else
> +#define _func_fmt "%s"
> +#define _func_arg ""
> +#endif
> +
> +extern int log_level;
> +extern bool log_timestamp;
> +#define _TIME_FMT "[%ld.%06ld] "
> +#define log(lvl, format, ...) \
> +	do {								\
> +		int __lvl = (lvl);					\
> +									\
> +		if (__lvl <= MAX_LOGLEVEL && __lvl <= log_level) {	\
> +			if (log_timestamp) {				\
> +				struct timespec __ts;			\
> +									\
> +				clock_gettime(CLOCK_MONOTONIC, &__ts);	\
> +				fprintf(stderr,				\
> +					_TIME_FMT _func_fmt format,	\
> +					__ts.tv_sec, __ts.tv_nsec / 1000,\
> +					_func_arg,			\
> +					##__VA_ARGS__);			\
> +			} else {					\
> +				fprintf(stderr, _func_fmt format,	\
> +					_func_arg,			\
> +					##__VA_ARGS__);			\
> +			};						\
> +		}							\
> +	} while (0)
> +
> +#endif /* _LOG_H */
> 
Urgh. Long macros are always horrible.
Can't you convert it into a function?

It might also be an idea to move this as the first function, as it's
arguably an extension to existing functionality, and not directly 
related to the monitor.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 07/35] fabrics: use log() macro

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Fabrics functionality will be used by the monitor code, we
> need consistent logging. This patch causes no functional change
> for the fabrics module.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   fabrics.c | 83 ++++++++++++++++++++++++++++++-------------------------
>   1 file changed, 45 insertions(+), 38 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 08/35] monitor: add command line options to control logging

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Standard syslog loglevels are used.
> Options:
> 
>   -S/--silent   to suppress LOG_NOTICE
>   -v/--verbose  to enable LOG_INFO (overrides -S)
>   -D/--debug    to enable LOG_DEBUG (overrides -S, -v)
>   -C/--clockstamps to enable time stamps on log messages.
> 
> I tried to use short options that don't conflict with options
> from nvme connect-all, because many of those options might be
> useful for the monitor, too.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 47 +++++++++++++++++++++++++++++++++++++++++------
>   1 file changed, 41 insertions(+), 6 deletions(-)
> ... and if you had moved the logging functionality prior to the monitor 
code, this patch could be merged with the main monitor functionality, too.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 09/35] nvme_get_ctrl_attr(): constify "path" argument

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   nvme-topology.c | 2 +-
>   nvme.h          | 2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/nvme-topology.c b/nvme-topology.c
> index 71371c5..f62446b 100644
> --- a/nvme-topology.c
> +++ b/nvme-topology.c
> @@ -45,7 +45,7 @@ close_fd:
>   	return subsysnqn;
>   }
>   
> -char *nvme_get_ctrl_attr(char *path, const char *attr)
> +char *nvme_get_ctrl_attr(const char *path, const char *attr)
>   {
>   	char *attrpath, *value;
>   	ssize_t ret;
> diff --git a/nvme.h b/nvme.h
> index 3fb1060..382e625 100644
> --- a/nvme.h
> +++ b/nvme.h
> @@ -109,7 +109,7 @@ int scan_subsystems(struct nvme_topology *t, const char *subsysnqn,
>   		    __u32 ns_instance, char *dev_dir);
>   void free_topology(struct nvme_topology *t);
>   char *get_nvme_subsnqn(char *path);
> -char *nvme_get_ctrl_attr(char *path, const char *attr);
> +char *nvme_get_ctrl_attr(const char *path, const char *attr);
>   
>   void *nvme_alloc(size_t len, bool *huge);
>   void nvme_free(void *p, bool huge);
> 
Could be moved prior to the monitor code as it's unrelated to that.

Other than that:

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 10/35] fabrics: export do_discover(), build_options() and config

[Hannes Reinecke]

On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> These functions will be called by the monitor code.
> Also export BUF_SIZE as the size of argstr.
> 
> Being able to access struct config and the "cfg" variable
> from fabrics.c is essential for the monitor to leverage the existing,
> well tested code as much as possible.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   fabrics.c | 33 ++++-----------------------------
>   fabrics.h | 34 ++++++++++++++++++++++++++++++++++
>   2 files changed, 38 insertions(+), 29 deletions(-)
> 
> diff --git a/fabrics.c b/fabrics.c
> index 555b6b4..c1a4bb5 100644
> --- a/fabrics.c
> +++ b/fabrics.c
> @@ -69,31 +69,7 @@ const char *conarg_traddr = "traddr";
>   const char *conarg_trsvcid = "trsvcid";
>   const char *conarg_host_traddr = "host_traddr";
>   
> -static struct config {
> -	char *nqn;
> -	char *transport;
> -	char *traddr;
> -	char *trsvcid;
> -	char *host_traddr;
> -	char *hostnqn;
> -	char *hostid;
> -	int  nr_io_queues;
> -	int  nr_write_queues;
> -	int  nr_poll_queues;
> -	int  queue_size;
> -	int  keep_alive_tmo;
> -	int  reconnect_delay;
> -	int  ctrl_loss_tmo;
> -	int  tos;
> -	char *raw;
> -	char *device;
> -	int  duplicate_connect;
> -	int  disable_sqflow;
> -	int  hdr_digest;
> -	int  data_digest;
> -	bool persistent;
> -	bool matching_only;
> -} cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
> +struct config cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
>   
>   struct connect_args {
>   	char *subsysnqn;
> @@ -107,7 +83,6 @@ struct connect_args {
>   
>   struct connect_args *tracked_ctrls;
>   
> -#define BUF_SIZE		4096
>   #define PATH_NVME_FABRICS	"/dev/nvme-fabrics"
>   #define PATH_NVMF_DISC		"/etc/nvme/discovery.conf"
>   #define PATH_NVMF_HOSTNQN	"/etc/nvme/hostnqn"
> @@ -226,7 +201,7 @@ static const char *cms_str(__u8 cm)
>   	return arg_str(cms, ARRAY_SIZE(cms), cm);
>   }
>   
> -static int do_discover(char *argstr, bool connect);
> +int do_discover(char *argstr, bool connect);
>   
>   /*
>    * parse strings with connect arguments to find a particular field.
> @@ -855,7 +830,7 @@ add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
>   	return 0;
>   }
>   
> -static int build_options(char *argstr, int max_len, bool discover)
> +int build_options(char *argstr, int max_len, bool discover)
>   {
>   	int len;
>   
> @@ -1264,7 +1239,7 @@ static void nvmf_get_host_identifiers(int ctrl_instance)
>   	cfg.hostid = nvme_get_ctrl_attr(path, "hostid");
>   }
>   
> -static int do_discover(char *argstr, bool connect)
> +int do_discover(char *argstr, bool connect)
>   {
>   	struct nvmf_disc_rsp_page_hdr *log = NULL;
>   	char *dev_name;
> diff --git a/fabrics.h b/fabrics.h
> index f5b8eaf..ce965a3 100644
> --- a/fabrics.h
> +++ b/fabrics.h
> @@ -10,4 +10,38 @@ extern int fabrics_connect(const char *desc, int argc, char **argv);
>   extern int fabrics_disconnect(const char *desc, int argc, char **argv);
>   extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);
>   
> +/* Symbols used by monitor.c */
> +
> +struct config {
> +	char *nqn;
> +	char *transport;
> +	char *traddr;
> +	char *trsvcid;
> +	char *host_traddr;
> +	char *hostnqn;
> +	char *hostid;
> +	int  nr_io_queues;
> +	int  nr_write_queues;
> +	int  nr_poll_queues;
> +	int  queue_size;
> +	int  keep_alive_tmo;
> +	int  reconnect_delay;
> +	int  ctrl_loss_tmo;
> +	int  tos;
> +	char *raw;
> +	char *device;
> +	int  duplicate_connect;
> +	int  disable_sqflow;
> +	int  hdr_digest;
> +	int  data_digest;
> +	bool persistent;
> +	bool matching_only;
> +};
> +extern struct config cfg;
> +
> +#define BUF_SIZE 4096
> +
> +int build_options(char *argstr, int max_len, bool discover);
> +int do_discover(char *argstr, bool connect);
> +
>   #endif
> 

Please name is 'struct config fabrics_cfg'. 'cfg' is used as a generic 
name throughout the code, and using the same name in a header might lead 
to an accidental name clash.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 11/35] monitor: add option -A / --autoconnect

[Hannes Reinecke]

On 1/26/21 9:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> With this option, the monitor will try to connect to newly discovered
> controllers.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/monitor.c b/monitor.c
> index a7afa1c..ecf3be2 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -31,6 +31,10 @@
>   #define LOG_FUNCNAME 1
>   #include "log.h"
>   
> +static struct monitor_config {
> +	bool autoconnect;
> +} mon_cfg;
> +
>   static struct udev *udev;
>   
>   static void close_ptr(int *p)
> @@ -177,6 +181,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
>   	bool debug = false;
>   	int ret;
>   	OPT_ARGS(opts) = {
> +		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
>   		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
>   		OPT_FLAG("verbose",        'v', &verbose,             "log level: verbose"),
>   		OPT_FLAG("debug",          'D', &debug,               "log level: debug"),
> 
This patch is slightly confusing.

It just adds an option, but there is not code evaluating this option. So 
where does it hook in?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 12/35] monitor: add helpers for __attribute__((cleanup))

[Hannes Reinecke]

On 1/26/21 9:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> __attribute__((cleanup)) is very helpful but ugly. Try to avoid
> defining lots of cleanup functions with these macros.
> 
> Usage: to declare an auto-cleanup variable of type (some_type *),
> write
> 
> CLEANUP_FUNC(some_type)
> 
> void some_func(void)
> {
>          CLEANUP(some_type, varname) = NULL;
> 	...
> }
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   common.h | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
> 
> diff --git a/common.h b/common.h
> index 1c214a4..9386ec5 100644
> --- a/common.h
> +++ b/common.h
> @@ -12,4 +12,16 @@
>   #define __stringify_1(x...) #x
>   #define __stringify(x...)  __stringify_1(x)
>   
> +#define CLEANUP_FUNC(type) \
> +static void __cleanup_ ## type ##_p(type ** __p) \
> +{						 \
> +	if (*__p) {				 \
> +		free(*__p);			 \
> +		*__p = NULL;			 \
> +	}					 \
> +}
> +
> +#define CLEANUP(__t, __v) \
> +	__t *__v __attribute__((cleanup(__cleanup_ ## __t ## _p)))
> +
>   #endif
> 
And this is used where?

Can't you merge it with the patch usig the __cleanup functionality?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[Hannes Reinecke]

On 1/26/21 9:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> If autoconnect is enabled, disable the respective udev rules
> by symlinking /run/udev/rules.d to /dev/null, in order to avoid
> the connections being set up by the monitor and the udev workers
> at the same time. This is probably the preferred mode of operation
> for the monitor.
> 
> Users can override this by copying 70-nvmf-autoconnect.rules
> from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
> takes precedence over /run/udev/rules.d).
> 
> If the symlink can't be created for some reason, autoconnect will
> be disabled. There is  only one exception: If
> /run/udev/rules.d/70-nvmf-autoconnect.rules already points to
> /dev/null at startup, autoconnect can be left on, but the symlink
> isn't removed on exit.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   monitor.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 73 insertions(+), 2 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index ecf3be2..2a906db 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -17,14 +17,18 @@
>   
>   #include <stddef.h>
>   #include <stdio.h>
> +#include <stdlib.h>
>   #include <unistd.h>
>   #include <errno.h>
>   #include <libudev.h>
>   #include <signal.h>
>   #include <time.h>
> +#include <limits.h>
>   #include <syslog.h>
> +#include <sys/stat.h>
>   #include <sys/epoll.h>
>   
> +#include "common.h"
>   #include "nvme-status.h"
>   #include "util/argconfig.h"
>   #include "monitor.h"
> @@ -33,6 +37,7 @@
>   
>   static struct monitor_config {
>   	bool autoconnect;
> +	bool skip_udev_on_exit;
>   } mon_cfg;
>   
>   static struct udev *udev;
> @@ -45,6 +50,8 @@ static void close_ptr(int *p)
>   	}
>   }
>   
> +CLEANUP_FUNC(char)
> +
>   static void cleanup_monitor(struct udev_monitor **pmon)
>   {
>   	if (*pmon) {
> @@ -174,12 +181,64 @@ static int monitor_main_loop(struct udev_monitor *monitor)
>   	return ret;
>   }
>   
> +static const char autoconnect_rules[] = "/run/udev/rules.d/70-nvmf-autoconnect.rules";
> +
> +static int monitor_disable_udev_rules(void)
> +{
> +	CLEANUP(char, path) = strdup(autoconnect_rules);
> +	char *s1, *s2;
> +	int rc;
> +
> +	if (!path)
> +		return -ENOMEM;
> +
> +	s2 = strrchr(path, '/');
> +	for (s1 = s2 - 1; s1 > path && *s1 != '/'; s1--);
> +
> +	*s2 = *s1 = '\0';
> +	rc = mkdir(path, 0755);
> +	if (rc == 0 || errno == EEXIST) {
> +		*s1 = '/';
> +		rc = mkdir(path, 0755);
> +		if (rc == 0 || errno == EEXIST) {
> +			*s2 = '/';
> +			rc = symlink("/dev/null", path);
> +		}
> +	}
> +	if (rc) {
> +		if (errno == EEXIST) {
> +			char target[PATH_MAX];
> +
> +			if (readlink(path, target, sizeof(target)) != -1 &&
> +			    !strcmp(target, "/dev/null")) {
> +				log(LOG_INFO,
> +				    "symlink %s -> /dev/null exists already\n",
> +				    autoconnect_rules);
> +				return 1;
> +			}
> +		}
> +		log(LOG_ERR, "error creating %s: %m\n", autoconnect_rules);
> +	} else
> +		log(LOG_INFO, "created %s\n", autoconnect_rules);
> +
> +	return rc ? (errno ? -errno : -EIO) : 0;
> +}
> +
> +static void monitor_enable_udev_rules(void)
> +{
> +	if (unlink(autoconnect_rules) == -1 && errno != ENOENT)
> +		log(LOG_ERR, "error removing %s: %m\n", autoconnect_rules);
> +	else
> +		log(LOG_INFO, "removed %s\n", autoconnect_rules);
> +}
> +
>   static int monitor_parse_opts(const char *desc, int argc, char **argv)
>   {
>   	bool quiet = false;
>   	bool verbose = false;
>   	bool debug = false;
> -	int ret;
> +	int ret = 0;
> +
>   	OPT_ARGS(opts) = {
>   		OPT_FLAG("autoconnect",    'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
>   		OPT_FLAG("silent",         'S', &quiet,               "log level: silent"),
> @@ -198,7 +257,17 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)
>   		log_level = LOG_INFO;
>   	if (debug)
>   		log_level = LOG_DEBUG;
> -
> +	if (mon_cfg.autoconnect) {
> +		ret = monitor_disable_udev_rules();
> +		if (ret < 0) {
> +			mon_cfg.autoconnect = false;
> +			log(LOG_WARNING, "autoconnect disabled\n");
> +			ret = 0;
> +		} else if (ret > 0) {
> +			mon_cfg.skip_udev_on_exit = true;
> +			ret = 0;
> +		}
> +	}
>   	return ret;
>   }
>   
> @@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc, char **argv)
>   		udev_monitor_unref(monitor);
>   	}
>   	udev = udev_unref(udev);
> +	if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
> +		monitor_enable_udev_rules();
>   out:
>   	return nvme_status_to_errno(ret, true);
>   }
> 
What on earth ...

No way.
We should _not_ reference individual files or mess with udev rules from 
a program. If a file needs to be modified, modify it.
But do not try to modify a file from a program context, unless the 
program itself has created it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 19/35] fabrics: use "const char *" in struct config

[Hannes Reinecke]

On 1/26/21 9:33 PM, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> This is easily done, and allows passing in some const char* pointers
> that would otherwise need to be strdup()d first.
> 
> constifying cfg->device requires changes in the ctrl_instance()
> function, because basename() can't be used.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>   fabrics.c | 23 ++++++++++++++---------
>   fabrics.h | 17 +++++++++--------
>   2 files changed, 23 insertions(+), 17 deletions(-)
> 
> diff --git a/fabrics.c b/fabrics.c
> index c1a4bb5..e3d2a3a 100644
> --- a/fabrics.c
> +++ b/fabrics.c
> @@ -252,17 +252,22 @@ empty_field:
>   	return strdup("\0");
>   }
>   
> -static int ctrl_instance(char *device)
> +int ctrl_instance(const char *device)
>   {
>   	char d[64];
> +	const char *p;
>   	int ret, instance;
>   
> -	device = basename(device);
> -	ret = sscanf(device, "nvme%d", &instance);
> +	p = strrchr(device, '/');
> +	if (p == NULL)
> +		p = device;
> +	else
> +		p++;
> +	ret = sscanf(p, "nvme%d", &instance);
>   	if (ret <= 0)
>   		return -EINVAL;
>   	if (snprintf(d, sizeof(d), "nvme%d", instance) <= 0 ||
> -	    strcmp(device, d))
> +	    strcmp(p, d))
>   		return -EINVAL;
>   	return instance;
>   }
> @@ -273,7 +278,7 @@ static int ctrl_instance(char *device)
>    * given.
>    * Return true/false based on whether it matches
>    */
> -static bool ctrl_matches_connectargs(char *name, struct connect_args *args)
> +static bool ctrl_matches_connectargs(const char *name, struct connect_args *args)
>   {
>   	struct connect_args cargs;
>   	bool found = false;
> @@ -815,7 +820,7 @@ add_int_argument(char **argstr, int *max_len, char *arg_str, int arg,
>   }
>   
>   static int
> -add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
> +add_argument(char **argstr, int *max_len, char *arg_str, const char *arg)
>   {
>   	int len;
>   
> @@ -1541,7 +1546,7 @@ static int scan_sys_nvme_filter(const struct dirent *d)
>   /*
>    * Returns 1 if disconnect occurred, 0 otherwise.
>    */
> -static int disconnect_subsys(char *nqn, char *ctrl)
> +static int disconnect_subsys(const char *nqn, char *ctrl)
>   {
>   	char *sysfs_nqn_path = NULL, *sysfs_del_path = NULL;
>   	char subsysnqn[NVMF_NQN_SIZE] = {};
> @@ -1579,7 +1584,7 @@ static int disconnect_subsys(char *nqn, char *ctrl)
>   /*
>    * Returns the number of controllers successfully disconnected.
>    */
> -static int disconnect_by_nqn(char *nqn)
> +static int disconnect_by_nqn(const char *nqn)
>   {
>   	struct dirent **devices = NULL;
>   	int i, n, ret = 0;
> @@ -1601,7 +1606,7 @@ static int disconnect_by_nqn(char *nqn)
>   	return ret;
>   }
>   
> -static int disconnect_by_device(char *device)
> +static int disconnect_by_device(const char *device)
>   {
>   	int instance;
>   
> diff --git a/fabrics.h b/fabrics.h
> index ce965a3..1dfbd67 100644
> --- a/fabrics.h
> +++ b/fabrics.h
> @@ -13,13 +13,13 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);
>   /* Symbols used by monitor.c */
>   
>   struct config {
> -	char *nqn;
> -	char *transport;
> -	char *traddr;
> -	char *trsvcid;
> -	char *host_traddr;
> -	char *hostnqn;
> -	char *hostid;
> +	const char *nqn;
> +	const char *transport;
> +	const char *traddr;
> +	const char *trsvcid;
> +	const char *host_traddr;
> +	const char *hostnqn;
> +	const char *hostid;
>   	int  nr_io_queues;
>   	int  nr_write_queues;
>   	int  nr_poll_queues;
> @@ -29,7 +29,7 @@ struct config {
>   	int  ctrl_loss_tmo;
>   	int  tos;
>   	char *raw;
> -	char *device;
> +	const char *device;
>   	int  duplicate_connect;
>   	int  disable_sqflow;
>   	int  hdr_digest;
> @@ -43,5 +43,6 @@ extern struct config cfg;
>   
>   int build_options(char *argstr, int max_len, bool discover);
>   int do_discover(char *argstr, bool connect);
> +int ctrl_instance(const char *device);
>   
>   #endif
> 
This really is independent on the monitor code, and should be moved to 
the beginning of the patch series.

Other than that:

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Hannes Reinecke]

On 1/29/21 10:15 PM, Martin Wilck wrote:
> On Fri, 2021-01-29 at 13:11 -0800, Sagi Grimberg wrote:
>>
>>>>> If it does, it will recreate discovery controllers for every
>>>>> host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics
>>>>> are
>>>>> only
>>>>> necessary for addresses on which no regular (non-discovery)
>>>>> connection
>>>>> exists.
>>>>
>>>> Wait, Maybe I'm missing something here, but are you saying that
>>>> for
>>>> every traddr/trsvcid it finds (both nvm and discovery) it will
>>>> attempt
>>>> to connect a discovery controller?
>>>>
>>>> If so, this is absolutely wrong.
>>>
>>> Currently, it tries to do that on startup, if (and only if) the
>>> --startup option is given. My expectation was that the connection
>>> attempts would simply fail if there was no discovery subsystem to
>>> connect to. Anyway, it's not the default behavior, and can be
>>> dropped
>>> completely if it's so bad that we shouldn't ever attempt to do it.
>>
>> IMO it needs to be dropped. I didn't understand this at first because
>> it never even occurred to me that such an assumption can be even
>> made.
>>
>>> If the service is started early on during boot, and event-based
>>> discovery works (i.e. we also have the mDNS part in place), this
>>> won't
>>> be necessary of course.
>>
>> This isn't necessary regardless. At best the discovery controller
>> endpoints should be obtained from discovery.conf or equivalent.
> 
> Understood.
> 
>> I don't even see how does this help in early boot anyways, how
>> do the existing controllers get connected?
> 
> For FC, it works today. You can start the monitor in the initrd, it
> will receive the fc_udev_device events, and autoconnect (just like
> traditional udev/systemd based discovery). For other transports, it'd
> be more work, as we'd need to do mDNS discovery from an initrd
> environment. I wouldn't say it's impossible though.
> 
mDNS isn't related to the initrd, mDNS is for discovery discovery :-)
For everything _but_ FC we have the problem that we don't know where to 
connect to for the initial discovery (ie we don't have a list of 
potential discovery controllers).
Currently we either have to specify the parameters manually, or add them 
to /etc/nvme/discovery.conf.
Which isn't _that_ dynamic.
mDNS will provide a way of promoting changes in any remote discovery 
controllers to the potential hosts, allowing the host to pick up 
information about potential discovery controllers from there.

So the monitor code would need to interact with mDNS to get the 
information, and should use that to establish discovery connections.

As this information is transient I would not tear down discovery 
controller connections on exit, but rather keep them alive.
That way a controller restart could pick up these existing connections 
and would not need to recreate them (which will be tricky as the monitor 
code doesn't have the information which connections to restart).

The alternative would be a callout to mDNS to get the current topology, 
but that doesn't exist yet.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Hannes Reinecke]

On 1/29/21 9:27 PM, Martin Wilck wrote:
> On Fri, 2021-01-29 at 12:08 -0800, Sagi Grimberg wrote:
>>
>>>>>
>>>>> This method for discovery and autodetection has some advantages
>>>>> over the
>>>>> current udev-rule based approach:
>>>>>
>>>>>     * By using the `--persistent` option, users can easily
>>>>> control
>>>>> whether
>>>>>       persistent discovery controllers for discovered transport
>>>>> addresses should
>>>>>       be created and monitored for AEN events. **nvme monitor**
>>>>> watches known
>>>>>       transport addresses, creates discovery controllers as
>>>>> required,
>>>>> and re-uses
>>>>>       existing ones if possible.
>>>>
>>>> What does that mean?
>>>
>>> In general, if the monitor detects a new host_traddr/traddr/trsvcid
>>> tuple, it runs a discovery on it, and keeps the discovery
>>> controller
>>> open if --persistent was given. On startup, it scans existing
>>> controllers, and if it finds already existing discovery
>>> controllers,
>>> re-uses them. These will not be shut down when the monitor exits.
>>
>> And if it doesn't run with --persistent it deletes them? even if
>> these
>> weren't created by it?
> 
> No, not those. That's what I wanted to express with "These will not be
> shut down when the monitor exits."
> 
> It's not trivial to determine reliably which controller the monitor
> created and which it didn't. The current code assumes that all
> discovery controllers that didn't exist at startup were created by the
> monitor. We can improve the intelligence of the tool in that area, of
> course. Currently it makes this dumb assumption.
> 
>>
>> And, if none exist where does it get new discovery controller
>> details?
>> Today we use discovery.conf for that.
> 
> As I said, this can be done with a separate "nvme connect-all" call,
> which could be run from an ExecStartPre or from a separate service.
> Or we can integrate the functionality in the monitor. I don't have a
> strong opinion either way. I can adapt to what you guys prefer.
> 
>>
>>> This allows users fine-grained control about what discovery
>>> controllers
>>> to operate persistently. Users who want all discovery controllers
>>> to be
>>> persistent just use --persistent. Others can set up those that they
>>> want to have (manually or with a script), and not use --persistent.
>>>
>>> The background is that hosts may not need every detected discovery
>>> controller to be persistent. In multipath scenarios, you may see
>>> more
>>> discovery subsystems than anything else, and not everyone likes
>>> that.
>>> That's a generic issue and unrelated to the monitor, but running
>>> the
>>> monitor with --persistent creates discovery controllers that would
>>> otherwise not be visible.
>>>
>>> Hope this clarifies it.
>>
>> Well, How do people expect to know if stuff change without a
>> persistent
>> discovery controller? Not sure if people may see more discovery
>> controllers this is a real issue given what they are getting from it.
> 
> You're right. At this very early stage, I wanted to give users the
> freedom of choice, and not force dozens of discovery controller
> connections upon everyone. If there's consensus that the discovery
> connections should always be created, fine with me.
> 
> 
I would make the monitor code to always create persistent discovery 
connections.
(Or, at least, _attempt_ to create persistent discovery connections.)

_if_ the user starts the monitor he will have a reason for it; and that 
reason typically is that he _wants_ the system to react on discovery 
changes.
And that is precisely why persistent discovery controllers exist.

Plus there is a provision in the spec allowing the controller to reject 
persistent connections, so if the controller is worried about increased 
resource consumption he can just disallow persistent controller connections.

Similarly, I would keep the created discovery connections after exit.
Currently the simple reason is that we don't store the information from 
the created discovery connections, so if we tear them down on exit we 
won't be able to recreate them once we restart the monitor.

This situation might change with once we have mDNS integration; and, of 
course, for FC figuring out this information is trivial.

>>> I agree. But it's not easy to fix the issue otherwise. In the
>>> customer
>>> problem where we observed it, I worked around it by adding the udev
>>> seqnum to the "instance name" of the systemd service, thus allowing
>>> several "nvme connect-all" processes to run for the same transport
>>> address simultaneously. But I don't think that would scale well;
>>> the monitor can handle it more cleanly.
>>
>> Still changing the entire thing for a corner case...
> 
> It's just one puzzle piece.
> 
And it's actually not a corner case, but a real issue we're seeing in 
our (or rather, our customers) deployments.
AENs are being received at the same time, causing 'nvme discover' to run 
in parallel with the same information.
A very nice exercise in finding all race conditions in the code.
And completely pointless to boot.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/35] monitor: Create a log() macro.

[Martin Wilck]

On Thu, 2021-02-04 at 08:01 +0100, Hannes Reinecke wrote:
> On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> > +               int __lvl =
> > (lvl);                                      \
> > +                                                                  
> >      \
> > +               if (__lvl <= MAX_LOGLEVEL && __lvl <= log_level)
> > {      \
> > +                       if (log_timestamp)
> > {                            \
> > +                               struct timespec
> > __ts;                   \
> > +                                                                  
> >      \
> > +                               clock_gettime(CLOCK_MONOTONIC,
> > &__ts);  \
> > +                               fprintf(stderr,                    
> >      \
> > +                                       _TIME_FMT _func_fmt
> > format,     \
> > +                                       __ts.tv_sec, __ts.tv_nsec /
> > 1000,\
> > +                                       _func_arg,                 
> >      \
> > +                                       ##__VA_ARGS__);            
> >      \
> > +                       } else
> > {                                        \
> > +                               fprintf(stderr, _func_fmt
> > format,       \
> > +                                       _func_arg,                 
> >      \
> > +                                       ##__VA_ARGS__);            
> >      \
> > +                       };                                         
> >      \
> > +               }                                                  
> >      \
> > +       } while (0)
> > +
> > +#endif /* _LOG_H */
> > 
> Urgh. Long macros are always horrible.
> Can't you convert it into a function?

Sure. It won't work without macros, but I can write smaller ones.

> It might also be an idea to move this as the first function, as it's
> arguably an extension to existing functionality, and not directly 
> related to the monitor.

Ok.

Thanks,
Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 08/35] monitor: add command line options to control logging

[Martin Wilck]

On Thu, 2021-02-04 at 08:04 +0100, Hannes Reinecke wrote:
> On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > Standard syslog loglevels are used.
> > Options:
> > 
> >   -S/--silent   to suppress LOG_NOTICE
> >   -v/--verbose  to enable LOG_INFO (overrides -S)
> >   -D/--debug    to enable LOG_DEBUG (overrides -S, -v)
> >   -C/--clockstamps to enable time stamps on log messages.
> > 
> > I tried to use short options that don't conflict with options
> > from nvme connect-all, because many of those options might be
> > useful for the monitor, too.
> > 
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > ---
> >   monitor.c | 47 +++++++++++++++++++++++++++++++++++++++++------
> >   1 file changed, 41 insertions(+), 6 deletions(-)
> > ... and if you had moved the logging functionality prior to the
> > monitor 
> code, this patch could be merged with the main monitor functionality,
> too.

I expected that reviewers preferred smaller chunks. If that's not the
case, fine with me, I'll change i

Thanks,
Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 10/35] fabrics: export do_discover(), build_options() and config

[Martin Wilck]

On Thu, 2021-02-04 at 08:09 +0100, Hannes Reinecke wrote:
> On 1/26/21 9:32 PM, mwilck@suse.com wrote:
> >   
> > +/* Symbols used by monitor.c */
> > +
> > +struct config {
> > +       char *nqn;
> > +       char *transport;
> > +       char *traddr;
> > +       char *trsvcid;
> > +       char *host_traddr;
> > +       char *hostnqn;
> > +       char *hostid;
> > +       int  nr_io_queues;
> > +       int  nr_write_queues;
> > +       int  nr_poll_queues;
> > +       int  queue_size;
> > +       int  keep_alive_tmo;
> > +       int  reconnect_delay;
> > +       int  ctrl_loss_tmo;
> > +       int  tos;
> > +       char *raw;
> > +       char *device;
> > +       int  duplicate_connect;
> > +       int  disable_sqflow;
> > +       int  hdr_digest;
> > +       int  data_digest;
> > +       bool persistent;
> > +       bool matching_only;
> > +};
> > +extern struct config cfg;
> > +
> > +#define BUF_SIZE 4096
> > +
> > +int build_options(char *argstr, int max_len, bool discover);
> > +int do_discover(char *argstr, bool connect);
> > +
> >   #endif
> > 
> 
> Please name is 'struct config fabrics_cfg'. 'cfg' is used as a
> generic 
> name throughout the code, and using the same name in a header might
> lead to an accidental name clash.

Will do (the reason I didn't in the first place was that I wanted to
minimize changes of existing code, but you're right of course).

Thanks,
Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode

[Martin Wilck]

On Thu, 2021-02-04 at 08:16 +0100, Hannes Reinecke wrote:
> On 1/26/21 9:33 PM, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > If autoconnect is enabled, disable the respective udev rules
> > by symlinking /run/udev/rules.d to /dev/null, in order to avoid
> > the connections being set up by the monitor and the udev workers
> > at the same time. This is probably the preferred mode of operation
> > for the monitor.
> > 
> > Users can override this by copying 70-nvmf-autoconnect.rules
> > from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d
> > takes precedence over /run/udev/rules.d).
> > 
> > If the symlink can't be created for some reason, autoconnect will
> > be disabled. There is  only one exception: If
> > /run/udev/rules.d/70-nvmf-autoconnect.rules already points to
> > /dev/null at startup, autoconnect can be left on, but the symlink
> > isn't removed on exit.
> > 
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > ---
> >   monitor.c | 75
> > +++++++++++++++++++++++++++++++++++++++++++++++++++++--
> >   1 file changed, 73 insertions(+), 2 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index ecf3be2..2a906db 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -17,14 +17,18 @@
> >   
> >   #include <stddef.h>
> >   #include <stdio.h>
> > +#include <stdlib.h>
> >   #include <unistd.h>
> >   #include <errno.h>
> >   #include <libudev.h>
> >   #include <signal.h>
> >   #include <time.h>
> > +#include <limits.h>
> >   #include <syslog.h>
> > +#include <sys/stat.h>
> >   #include <sys/epoll.h>
> >   
> > +#include "common.h"
> >   #include "nvme-status.h"
> >   #include "util/argconfig.h"
> >   #include "monitor.h"
> > @@ -33,6 +37,7 @@
> >   
> >   static struct monitor_config {
> >         bool autoconnect;
> > +       bool skip_udev_on_exit;
> >   } mon_cfg;
> >   
> >   static struct udev *udev;
> > @@ -45,6 +50,8 @@ static void close_ptr(int *p)
> >         }
> >   }
> >   
> > +CLEANUP_FUNC(char)
> > +
> >   static void cleanup_monitor(struct udev_monitor **pmon)
> >   {
> >         if (*pmon) {
> > @@ -174,12 +181,64 @@ static int monitor_main_loop(struct
> > udev_monitor *monitor)
> >         return ret;
> >   }
> >   
> > +static const char autoconnect_rules[] = "/run/udev/rules.d/70-
> > nvmf-autoconnect.rules";
> > +
> > +static int monitor_disable_udev_rules(void)
> > +{
> > +       CLEANUP(char, path) = strdup(autoconnect_rules);
> > +       char *s1, *s2;
> > +       int rc;
> > +
> > +       if (!path)
> > +               return -ENOMEM;
> > +
> > +       s2 = strrchr(path, '/');
> > +       for (s1 = s2 - 1; s1 > path && *s1 != '/'; s1--);
> > +
> > +       *s2 = *s1 = '\0';
> > +       rc = mkdir(path, 0755);
> > +       if (rc == 0 || errno == EEXIST) {
> > +               *s1 = '/';
> > +               rc = mkdir(path, 0755);
> > +               if (rc == 0 || errno == EEXIST) {
> > +                       *s2 = '/';
> > +                       rc = symlink("/dev/null", path);
> > +               }
> > +       }
> > +       if (rc) {
> > +               if (errno == EEXIST) {
> > +                       char target[PATH_MAX];
> > +
> > +                       if (readlink(path, target, sizeof(target))
> > != -1 &&
> > +                           !strcmp(target, "/dev/null")) {
> > +                               log(LOG_INFO,
> > +                                   "symlink %s -> /dev/null exists
> > already\n",
> > +                                   autoconnect_rules);
> > +                               return 1;
> > +                       }
> > +               }
> > +               log(LOG_ERR, "error creating %s: %m\n",
> > autoconnect_rules);
> > +       } else
> > +               log(LOG_INFO, "created %s\n", autoconnect_rules);
> > +
> > +       return rc ? (errno ? -errno : -EIO) : 0;
> > +}
> > +
> > +static void monitor_enable_udev_rules(void)
> > +{
> > +       if (unlink(autoconnect_rules) == -1 && errno != ENOENT)
> > +               log(LOG_ERR, "error removing %s: %m\n",
> > autoconnect_rules);
> > +       else
> > +               log(LOG_INFO, "removed %s\n", autoconnect_rules);
> > +}
> > +
> >   static int monitor_parse_opts(const char *desc, int argc, char
> > **argv)
> >   {
> >         bool quiet = false;
> >         bool verbose = false;
> >         bool debug = false;
> > -       int ret;
> > +       int ret = 0;
> > +
> >         OPT_ARGS(opts) = {
> >                 OPT_FLAG("autoconnect",    'A',
> > &mon_cfg.autoconnect, "automatically connect newly discovered
> > controllers"),
> >                 OPT_FLAG("silent",         'S',
> > &quiet,               "log level: silent"),
> > @@ -198,7 +257,17 @@ static int monitor_parse_opts(const char
> > *desc, int argc, char **argv)
> >                 log_level = LOG_INFO;
> >         if (debug)
> >                 log_level = LOG_DEBUG;
> > -
> > +       if (mon_cfg.autoconnect) {
> > +               ret = monitor_disable_udev_rules();
> > +               if (ret < 0) {
> > +                       mon_cfg.autoconnect = false;
> > +                       log(LOG_WARNING, "autoconnect disabled\n");
> > +                       ret = 0;
> > +               } else if (ret > 0) {
> > +                       mon_cfg.skip_udev_on_exit = true;
> > +                       ret = 0;
> > +               }
> > +       }
> >         return ret;
> >   }
> >   
> > @@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc,
> > char **argv)
> >                 udev_monitor_unref(monitor);
> >         }
> >         udev = udev_unref(udev);
> > +       if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
> > +               monitor_enable_udev_rules();
> >   out:
> >         return nvme_status_to_errno(ret, true);
> >   }
> > 
> What on earth ...
> 
> No way.
> We should _not_ reference individual files or mess with udev rules
> from 
> a program. If a file needs to be modified, modify it.
> But do not try to modify a file from a program context, unless the 
> program itself has created it.

Sagi said the same, I'll drop this. I thought it was acceptable because
I'd be masking a rule from the same package, and preferable because
making the user responsible for it would be cumbersome and error-prone
on the part of the user.

Thanks,
Martin

> 
> Cheers,
> 
> Hannes



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 35/35] monitor: add option --keep/-K

[Martin Wilck]

On Thu, 2021-02-04 at 08:34 +0100, Hannes Reinecke wrote:
> On 1/29/21 10:15 PM, Martin Wilck wrote:
> > On Fri, 2021-01-29 at 13:11 -0800, Sagi Grimberg wrote:
> > > 
> > > > > > If it does, it will recreate discovery controllers for
> > > > > > every
> > > > > > host_traddr/traddr/trsvcid tuple it finds. "--keep"
> > > > > > semantics
> > > > > > are
> > > > > > only
> > > > > > necessary for addresses on which no regular (non-discovery)
> > > > > > connection
> > > > > > exists.
> > > > > 
> > > > > Wait, Maybe I'm missing something here, but are you saying
> > > > > that
> > > > > for
> > > > > every traddr/trsvcid it finds (both nvm and discovery) it
> > > > > will
> > > > > attempt
> > > > > to connect a discovery controller?
> > > > > 
> > > > > If so, this is absolutely wrong.
> > > > 
> > > > Currently, it tries to do that on startup, if (and only if) the
> > > > --startup option is given. My expectation was that the
> > > > connection
> > > > attempts would simply fail if there was no discovery subsystem
> > > > to
> > > > connect to. Anyway, it's not the default behavior, and can be
> > > > dropped
> > > > completely if it's so bad that we shouldn't ever attempt to do
> > > > it.
> > > 
> > > IMO it needs to be dropped. I didn't understand this at first
> > > because
> > > it never even occurred to me that such an assumption can be even
> > > made.
> > > 
> > > > If the service is started early on during boot, and event-based
> > > > discovery works (i.e. we also have the mDNS part in place),
> > > > this
> > > > won't
> > > > be necessary of course.
> > > 
> > > This isn't necessary regardless. At best the discovery controller
> > > endpoints should be obtained from discovery.conf or equivalent.
> > 
> > Understood.
> > 
> > > I don't even see how does this help in early boot anyways, how
> > > do the existing controllers get connected?
> > 
> > For FC, it works today. You can start the monitor in the initrd, it
> > will receive the fc_udev_device events, and autoconnect (just like
> > traditional udev/systemd based discovery). For other transports,
> > it'd
> > be more work, as we'd need to do mDNS discovery from an initrd
> > environment. I wouldn't say it's impossible though.
> > 
> mDNS isn't related to the initrd, mDNS is for discovery discovery :-)

What I had in mind was booting from NVMeoF (non-FC), without prior
local configuration. That requires "discovery discovery" in the initrd,
and thus making mDNS calls in the initrd.

Regards
Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Enzo Matsumiya]

Hi,

On 01/26, mwilck@suse.com wrote:
>From: Martin Wilck <mwilck@suse.com>
>
>(Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)
>
>This patch set adds a new subcommand **nvme monitor**. In this mode,
>**nvme-cli** runs continuously, monitors events (currently, uevents) relevant
>for discovery, and optionally autoconnects to newly discovered subsystems.

Is there anything missing from this patch set?

> * Implement support for RDMA and TCP protocols.

I'm working on the RDMA and TCP implementations with mDNS/Avahi support, and
I would like to submit my work based on this monitor solution.


Cheers,

Enzo

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 00/35] RFC: add "nvme monitor" subcommand

[Martin Wilck]

On Mon, 2021-02-22 at 16:02 -0300, Enzo Matsumiya wrote:
> Hi,
> 
> On 01/26, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > (Cover letter copied from 
> > https://github.com/linux-nvme/nvme-cli/pull/877)
> > 
> > This patch set adds a new subcommand **nvme monitor**. In this
> > mode,
> > **nvme-cli** runs continuously, monitors events (currently,
> > uevents) relevant
> > for discovery, and optionally autoconnects to newly discovered
> > subsystems.
> 
> Is there anything missing from this patch set?
> 
> > * Implement support for RDMA and TCP protocols.
> 
> I'm working on the RDMA and TCP implementations with mDNS/Avahi
> support, and
> I would like to submit my work based on this monitor solution.

I'm still working on a v2. Please have some more patience.

Regards,
Martin



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme