[dm-devel] [PATCH v2 00/47] multipathd: uxlsnr overhaul

[dm-devel] [PATCH v2 00/47] multipathd: uxlsnr overhaul

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Hello Christophe, hello Ben,

The current multipathd unix listener code has various deficiencies.

 - client disconnects aren't handled correctly,
 - the uxsock_timeout is applied for receiving, handling, and
   responding to the client requests separately, rather than for
   the entire operation,
 - timeouts are logged, but not acted upon, causing the timeout
   to be noticed in the client rather than in the server.
 - clients may see a timeout while "reconfigure" is running,
 - unpriviledged (non-root) client connections don't work
   correctly
 - most importantly, the code busy-loops, polls, or waits in
   various places in called subroutines, which is a no-go in a
   piece of code designed as an event handler and may lead
   to spurious timeouts and delayed reaction e.g. to signals
   or client requests.

This patch set approaches all these issues. Fixing the last one,
in particular, requires a major refactoring of the uxlsnr code.
Overall, the reliability and latency of client request handling
and signal handling by multipathd should be noticeably improved
by this patch set.

The biggest problem (waiting for the vecs lock in a client handler)
can only be fixed by moving this wait into the handlers ppoll()
loop (another possible fix would have been to handle all clients
in separate threads, but that would have required even more
complexity). The patch set achieves this by adding an eventfd-based
notification mechanism to the vecs lock, which can be passed to
ppoll() to wake up when the lock is freed.

Furthermore, client requests can't be handled in a single poll
iteration any more. Therefore the client connection becomes stateful,
and is handled by a state machine using the states RECEIVE, PARSE,
WAIT FOR LOCK, WORK, and SEND.

The refactoring is done step by step for ease (hopefully) of
review. 1/35-4/35 add utility code that will be used by the uxlsnr
refactoring. 5/35-7/35 are some independent patches that
aren't directly related to uxlnsr, but fix issues that I observed
while working on this set.

8/35-13/35 are minor fixups in the client handling code. This code is
strongly related to the uxlsnr, thus I thought I'd rather fix it
before making the other changes. In 25/35, the cli-handlers are
converted to use the strbuf API everywhere instead of separate "reply"
and "len" arguments. 15/35-18/35 are minor fixes for the
uxlsnr. 19/35-34/35 are the actual refactoring patches for the uxlsnr
code. First I move some code around unchanged, then I add the
state machine (handle_client()) and move the code into it piece
by piece. 35/35 adds a fix for the client side (multipathd -k).

CC'ing Lixiaokeng and Chongyun Wu, as they have test cases that use
the client code heavily AFAIR. Testing by 3rd parties would be
very welcome.

---

Changes wrt v1 (Ben Marzinski):
  03: this is a major library version change.
  07: make set_config_state() static
  12: further simplify add_handler, make it static, and use assert
        to check for multiply-defined handlers
  14: dropped in favor of Ben's "reconfigure all" set, numbering changes
        from here on
  29 (was 30): don't use fallthrough; call state machine in a loop instead.
     fix signedness of return codes. Fix double messages.
  30 (was 31): The lock handling in this patch was broken. It could happen that
     the uxlsnr was cancelled without releasing the lock. Fixed by
     simplification. 
  35 (new): Use recv() for getting the command length, as suggested by Ben.

Moreover, I'm reposting Ben's rebased series on top of mine.

Comments welcome, regards,
Martin

Benjamin Marzinski (12):
  multipathd: move delayed_reconfig out of struct config
  multipathd: remove reconfigure from header file.
  multipathd: pass in the type of reconfigure
  multipathd: add "reconfigure all" command.
  multipathd: remove missing paths on startup
  libmultipath: skip unneeded steps to get path name
  libmultipath: don't use fallback wwid in update_pathvec_from_dm
  libmultipath: always set INIT_REMOVED in set_path_removed
  multipathd: fully initialize paths added by update_pathvec_from_dm
  multipathd: retrigger uevent for partial paths
  multipathd: remove INIT_PARTIAL paths that aren't in a multipath
    device
  multipathd: Remove dependency on systemd-udev-settle.service

Martin Wilck (36):
  libmultipath: add timespeccmp() utility function
  libmultipath: add trylock() helper
  libmultipath: add optional wakeup functionality to lock.c
  libmultipath: print: add __snprint_config()
  libmultipath: improve cleanup of uevent queues on exit
  multipathd: fix systemd notification when stopping while reloading
  multipathd: improve delayed reconfigure
  multipathd: cli.h: formatting improvements
  multipathd: cli_del_map: fix reply for delayed action
  multipathd: add prototype for cli_handler functions
  multipathd: make all cli_handlers static
  multipathd: add and set cli_handlers in a single step
  multipathd: cli.c: use ESRCH for "command not found"
  multipathd: uxlsnr: avoid stalled clients during reconfigure
  multipathd: uxlsnr: handle client HUP
  multipathd: uxlsnr: use symbolic values for pollfd indices
  multipathd: uxlsnr: avoid using fd -1 in ppoll()
  multipathd: uxlsnr: data structure for stateful client connection
  multipathd: move uxsock_trigger() to uxlsnr.c
  multipathd: move parse_cmd() to uxlsnr.c
  multipathd: uxlsnr: remove check_timeout()
  multipathd: uxlsnr: move client handling to separate function
  multipathd: uxlsnr: use main poll loop for receiving
  multipathd: use strbuf in cli_handler functions
  multipathd: uxlsnr: check root on connection startup
  multipathd: uxlsnr: pass struct client to uxsock_trigger() and
    parse_cmd()
  multipathd: uxlsnr: move handler execution to separate function
  multipathd: uxlsnr: use parser to determine non-root commands
  multipathd: uxlsnr: merge uxsock_trigger() into state machine
  multipathd: uxlsnr: add idle notification
  multipathd: uxlsnr: add timeout handling
  multipathd: uxlsnr: use poll loop for sending, too
  multipathd: uxlsnr: drop client_lock
  multipathd: uxclt: allow client mode for non-root, too
  multipathd: uxlsnr: use recv() for command length
  libmultipath: add path wildcard "%I" for init state

 libmpathpersist/libmpathpersist.version |  12 +-
 libmultipath/config.h                   |   1 -
 libmultipath/configure.c                |   4 +-
 libmultipath/devmapper.c                |   2 +
 libmultipath/discovery.c                |   7 +-
 libmultipath/discovery.h                |   2 +
 libmultipath/libmultipath.version       |  10 +-
 libmultipath/lock.c                     |  12 +-
 libmultipath/lock.h                     |  11 +-
 libmultipath/print.c                    |  55 ++-
 libmultipath/print.h                    |   2 +
 libmultipath/structs.h                  |   9 +
 libmultipath/structs_vec.c              |  41 +-
 libmultipath/structs_vec.h              |   2 +-
 libmultipath/time-util.c                |  12 +
 libmultipath/time-util.h                |   1 +
 libmultipath/uevent.c                   |  49 +-
 multipath/main.c                        |   2 +-
 multipathd/cli.c                        | 181 ++-----
 multipathd/cli.h                        | 102 ++--
 multipathd/cli_handlers.c               | 596 ++++++++++++------------
 multipathd/cli_handlers.h               |  61 +--
 multipathd/main.c                       | 357 +++++++-------
 multipathd/main.h                       |   4 +-
 multipathd/multipathd.8                 |  10 +-
 multipathd/multipathd.service           |   3 +-
 multipathd/uxlsnr.c                     | 542 +++++++++++++++------
 multipathd/uxlsnr.h                     |   4 +-
 28 files changed, 1193 insertions(+), 901 deletions(-)

-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 01/48] libmultipath: add timespeccmp() utility function

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Add a small utility that will be used in later patches.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/libmultipath.version |  5 +++++
 libmultipath/time-util.c          | 12 ++++++++++++
 libmultipath/time-util.h          |  1 +
 3 files changed, 18 insertions(+)

diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index eb5b5b5..c98cf7f 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -287,3 +287,8 @@ global:
 local:
 	*;
 };
+
+LIBMULTIPATH_9.1.0 {
+global:
+	timespeccmp;
+} LIBMULTIPATH_9.0.0;
diff --git a/libmultipath/time-util.c b/libmultipath/time-util.c
index 55f366c..2919300 100644
--- a/libmultipath/time-util.c
+++ b/libmultipath/time-util.c
@@ -49,3 +49,15 @@ void timespecsub(const struct timespec *a, const struct timespec *b,
 	res->tv_nsec = a->tv_nsec - b->tv_nsec;
 	normalize_timespec(res);
 }
+
+int timespeccmp(const struct timespec *a, const struct timespec *b)
+{
+	struct timespec tmp;
+
+	timespecsub(a, b, &tmp);
+	if (tmp.tv_sec > 0)
+		return 1;
+	if (tmp.tv_sec < 0)
+		return -1;
+	return tmp.tv_nsec > 0 ? 1 : (tmp.tv_nsec < 0 ? -1 : 0);
+}
diff --git a/libmultipath/time-util.h b/libmultipath/time-util.h
index b23d328..4a80ebd 100644
--- a/libmultipath/time-util.h
+++ b/libmultipath/time-util.h
@@ -10,5 +10,6 @@ void pthread_cond_init_mono(pthread_cond_t *cond);
 void normalize_timespec(struct timespec *ts);
 void timespecsub(const struct timespec *a, const struct timespec *b,
 		 struct timespec *res);
+int timespeccmp(const struct timespec *a, const struct timespec *b);
 
 #endif /* _TIME_UTIL_H_ */
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 02/48] libmultipath: add trylock() helper

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Add a small helper.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/lock.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/libmultipath/lock.h b/libmultipath/lock.h
index a170efe..d99eedb 100644
--- a/libmultipath/lock.h
+++ b/libmultipath/lock.h
@@ -12,6 +12,11 @@ static inline void lock(struct mutex_lock *a)
 	pthread_mutex_lock(&a->mutex);
 }
 
+static inline int trylock(struct mutex_lock *a)
+{
+	return pthread_mutex_trylock(&a->mutex);
+}
+
 static inline int timedlock(struct mutex_lock *a, struct timespec *tmo)
 {
 	return pthread_mutex_timedlock(&a->mutex, tmo);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Have struct mutex_lock take an optional wakeup function.
unlock() is renamed to __unlock() in order to prevent it from
being called by mistake.

This changes offsets in "struct vectors", requiring a major
libmultipath version bump. While the strucure is already changed,
in order to avoid this in the future, move the lock to the end
of "struct vectors".

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/libmultipath.version | 13 +++++++------
 libmultipath/lock.c               | 12 +++++++++++-
 libmultipath/lock.h               |  6 +++++-
 libmultipath/structs_vec.h        |  2 +-
 4 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index c98cf7f..98ec502 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -31,7 +31,7 @@
  *   The new version inherits the previous ones.
  */
 
-LIBMULTIPATH_9.0.0 {
+LIBMULTIPATH_10.0.0 {
 global:
 	/* symbols referenced by multipath and multipathd */
 	add_foreign;
@@ -284,11 +284,12 @@ global:
 	/* added in 8.2.0 */
 	check_daemon;
 
+	/* added in 9.1.0 */
+	timespeccmp;
+
+	/* added in 9.2.0 */
+	set_wakeup_fn;
+
 local:
 	*;
 };
-
-LIBMULTIPATH_9.1.0 {
-global:
-	timespeccmp;
-} LIBMULTIPATH_9.0.0;
diff --git a/libmultipath/lock.c b/libmultipath/lock.c
index 72c70e3..93b48db 100644
--- a/libmultipath/lock.c
+++ b/libmultipath/lock.c
@@ -3,6 +3,16 @@
 void cleanup_lock (void * data)
 {
 	struct mutex_lock *lock = data;
+	wakeup_fn *fn = lock->wakeup;
 
-	unlock(lock);
+	__unlock(lock);
+	if (fn)
+		fn();
+}
+
+void set_wakeup_fn(struct mutex_lock *lck, wakeup_fn *fn)
+{
+	lock(lck);
+	lck->wakeup = fn;
+	__unlock(lck);
 }
diff --git a/libmultipath/lock.h b/libmultipath/lock.h
index d99eedb..d7b779e 100644
--- a/libmultipath/lock.h
+++ b/libmultipath/lock.h
@@ -3,8 +3,11 @@
 
 #include <pthread.h>
 
+typedef void (wakeup_fn)(void);
+
 struct mutex_lock {
 	pthread_mutex_t mutex;
+	wakeup_fn *wakeup;
 };
 
 static inline void lock(struct mutex_lock *a)
@@ -22,7 +25,7 @@ static inline int timedlock(struct mutex_lock *a, struct timespec *tmo)
 	return pthread_mutex_timedlock(&a->mutex, tmo);
 }
 
-static inline void unlock(struct mutex_lock *a)
+static inline void __unlock(struct mutex_lock *a)
 {
 	pthread_mutex_unlock(&a->mutex);
 }
@@ -30,5 +33,6 @@ static inline void unlock(struct mutex_lock *a)
 #define lock_cleanup_pop(a) pthread_cleanup_pop(1)
 
 void cleanup_lock (void * data);
+void set_wakeup_fn(struct mutex_lock *lock, wakeup_fn *fn);
 
 #endif /* _LOCK_H */
diff --git a/libmultipath/structs_vec.h b/libmultipath/structs_vec.h
index 29ede45..2a0cbd1 100644
--- a/libmultipath/structs_vec.h
+++ b/libmultipath/structs_vec.h
@@ -6,9 +6,9 @@
 #include "lock.h"
 
 struct vectors {
-	struct mutex_lock lock; /* defined in lock.h */
 	vector pathvec;
 	vector mpvec;
+	struct mutex_lock lock; /* defined in lock.h */
 };
 
 void __set_no_path_retry(struct multipath *mpp, bool check_features);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 04/48] libmultipath: print: add __snprint_config()

[mwilck]

From: Martin Wilck <mwilck@suse.com>

exactly like snprint_config(), but takes a struct strbuf * as argument.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/libmultipath.version |  5 +++++
 libmultipath/print.c              | 34 +++++++++++++++++++++----------
 libmultipath/print.h              |  2 ++
 3 files changed, 30 insertions(+), 11 deletions(-)

diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index 98ec502..2fb2547 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -293,3 +293,8 @@ global:
 local:
 	*;
 };
+
+LIBMULTIPATH_10.1.0 {
+global:
+	__snprint_config;
+} LIBMULTIPATH_10.0.0;
diff --git a/libmultipath/print.c b/libmultipath/print.c
index 2fb9f4e..d2ef010 100644
--- a/libmultipath/print.c
+++ b/libmultipath/print.c
@@ -1756,24 +1756,36 @@ static int snprint_blacklist_except(const struct config *conf,
 	return get_strbuf_len(buff) - initial_len;
 }
 
+int __snprint_config(const struct config *conf, struct strbuf *buff,
+		     const struct _vector *hwtable, const struct _vector *mpvec)
+{
+	int rc;
+
+	if ((rc = snprint_defaults(conf, buff)) < 0 ||
+	    (rc = snprint_blacklist(conf, buff)) < 0 ||
+	    (rc = snprint_blacklist_except(conf, buff)) < 0 ||
+	    (rc = snprint_hwtable(conf, buff,
+				  hwtable ? hwtable : conf->hwtable)) < 0 ||
+	    (rc = snprint_overrides(conf, buff, conf->overrides)) < 0)
+		return rc;
+
+	if (VECTOR_SIZE(conf->mptable) > 0 ||
+	    (mpvec != NULL && VECTOR_SIZE(mpvec) > 0))
+		if ((rc = snprint_mptable(conf, buff, mpvec)) < 0)
+			return rc;
+
+	return 0;
+}
+
 char *snprint_config(const struct config *conf, int *len,
 		     const struct _vector *hwtable, const struct _vector *mpvec)
 {
 	STRBUF_ON_STACK(buff);
 	char *reply;
-	int rc;
+	int rc = __snprint_config(conf, &buff, hwtable, mpvec);
 
-	if ((rc = snprint_defaults(conf, &buff)) < 0 ||
-	    (rc = snprint_blacklist(conf, &buff)) < 0 ||
-	    (rc = snprint_blacklist_except(conf, &buff)) < 0 ||
-	    (rc = snprint_hwtable(conf, &buff,
-				  hwtable ? hwtable : conf->hwtable)) < 0 ||
-	    (rc = snprint_overrides(conf, &buff, conf->overrides)) < 0)
+	if (rc < 0)
 		return NULL;
-	if (VECTOR_SIZE(conf->mptable) > 0 ||
-	    (mpvec != NULL && VECTOR_SIZE(mpvec) > 0))
-		if ((rc = snprint_mptable(conf, &buff, mpvec)) < 0)
-			return NULL;
 
 	if (len)
 		*len = get_strbuf_len(&buff);
diff --git a/libmultipath/print.h b/libmultipath/print.h
index c6674a5..b149275 100644
--- a/libmultipath/print.h
+++ b/libmultipath/print.h
@@ -54,6 +54,8 @@ int _snprint_multipath_topology (const struct gen_multipath *, struct strbuf *,
 #define snprint_multipath_topology(buf, mpp, v) \
 	_snprint_multipath_topology (dm_multipath_to_gen(mpp), buf, v)
 int snprint_multipath_topology_json(struct strbuf *, const struct vectors *vecs);
+int __snprint_config(const struct config *conf, struct strbuf *buff,
+		     const struct _vector *hwtable, const struct _vector *mpvec);
 char *snprint_config(const struct config *conf, int *len,
 		     const struct _vector *hwtable,
 		     const struct _vector *mpvec);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 05/48] libmultipath: improve cleanup of uevent queues on exit

[mwilck]

From: Martin Wilck <mwilck@suse.com>

uevents listed on merge_node must be cleaned up, too. uevents
cancelled while being serviced and temporary queues, likewise.
The global uevq must be cleaned out in the uevent listener thread,
because it might have added events after the dispatcher thread
had already finished.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/uevent.c | 49 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 12 deletions(-)

diff --git a/libmultipath/uevent.c b/libmultipath/uevent.c
index 4265904..082e891 100644
--- a/libmultipath/uevent.c
+++ b/libmultipath/uevent.c
@@ -91,16 +91,25 @@ struct uevent * alloc_uevent (void)
 	return uev;
 }
 
+static void uevq_cleanup(struct list_head *tmpq);
+
+static void cleanup_uev(void *arg)
+{
+	struct uevent *uev = arg;
+
+	uevq_cleanup(&uev->merge_node);
+	if (uev->udev)
+		udev_device_unref(uev->udev);
+	FREE(uev);
+}
+
 static void uevq_cleanup(struct list_head *tmpq)
 {
 	struct uevent *uev, *tmp;
 
 	list_for_each_entry_safe(uev, tmp, tmpq, node) {
 		list_del_init(&uev->node);
-
-		if (uev->udev)
-			udev_device_unref(uev->udev);
-		FREE(uev);
+		cleanup_uev(uev);
 	}
 }
 
@@ -384,14 +393,10 @@ service_uevq(struct list_head *tmpq)
 	list_for_each_entry_safe(uev, tmp, tmpq, node) {
 		list_del_init(&uev->node);
 
+		pthread_cleanup_push(cleanup_uev, uev);
 		if (my_uev_trigger && my_uev_trigger(uev, my_trigger_data))
 			condlog(0, "uevent trigger error");
-
-		uevq_cleanup(&uev->merge_node);
-
-		if (uev->udev)
-			udev_device_unref(uev->udev);
-		FREE(uev);
+		pthread_cleanup_pop(1);
 	}
 }
 
@@ -411,6 +416,18 @@ static void monitor_cleanup(void *arg)
 	udev_monitor_unref(monitor);
 }
 
+static void cleanup_uevq(void *arg)
+{
+	uevq_cleanup(arg);
+}
+
+static void cleanup_global_uevq(void *arg __attribute__((unused)))
+{
+	pthread_mutex_lock(uevq_lockp);
+	uevq_cleanup(&uevq);
+	pthread_mutex_unlock(uevq_lockp);
+}
+
 /*
  * Service the uevent queue.
  */
@@ -425,6 +442,7 @@ int uevent_dispatch(int (*uev_trigger)(struct uevent *, void * trigger_data),
 	while (1) {
 		LIST_HEAD(uevq_tmp);
 
+		pthread_cleanup_push(cleanup_mutex, uevq_lockp);
 		pthread_mutex_lock(uevq_lockp);
 		servicing_uev = 0;
 		/*
@@ -436,14 +454,17 @@ int uevent_dispatch(int (*uev_trigger)(struct uevent *, void * trigger_data),
 		}
 		servicing_uev = 1;
 		list_splice_init(&uevq, &uevq_tmp);
-		pthread_mutex_unlock(uevq_lockp);
+		pthread_cleanup_pop(1);
+
 		if (!my_uev_trigger)
 			break;
+
+		pthread_cleanup_push(cleanup_uevq, &uevq_tmp);
 		merge_uevq(&uevq_tmp);
 		service_uevq(&uevq_tmp);
+		pthread_cleanup_pop(1);
 	}
 	condlog(3, "Terminating uev service queue");
-	uevq_cleanup(&uevq);
 	return 0;
 }
 
@@ -600,6 +621,8 @@ int uevent_listen(struct udev *udev)
 
 	events = 0;
 	gettimeofday(&start_time, NULL);
+	pthread_cleanup_push(cleanup_global_uevq, NULL);
+	pthread_cleanup_push(cleanup_uevq, &uevlisten_tmp);
 	while (1) {
 		struct uevent *uev;
 		struct udev_device *dev;
@@ -650,6 +673,8 @@ int uevent_listen(struct udev *udev)
 		gettimeofday(&start_time, NULL);
 		timeout = 30;
 	}
+	pthread_cleanup_pop(1);
+	pthread_cleanup_pop(1);
 out:
 	pthread_cleanup_pop(1);
 out_udev:
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 06/48] multipathd: fix systemd notification when stopping while reloading

[mwilck]

From: Martin Wilck <mwilck@suse.com>

After sending "RELOADING=1" to systemd, a service must send
"READY=1" before "STOPPING=1". Otherwise systemd will be confused
and will not regard the service as stopped. Subsequent attempts
to start multipathd via socket activation fail until systemd times
out the reload operation.

The problem can be reproduced by running "multipathd shutdown"
quickly after "multipathd reconfigure".

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/main.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 1defeaf..1a1812e 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -210,9 +210,12 @@ static void do_sd_notify(enum daemon_status old_state,
 	if (msg && !safe_sprintf(notify_msg, "STATUS=%s", msg))
 		sd_notify(0, notify_msg);
 
-	if (new_state == DAEMON_SHUTDOWN)
+	if (new_state == DAEMON_SHUTDOWN) {
+		/* Tell systemd that we're not RELOADING any more */
+		if (old_state == DAEMON_CONFIGURE && startup_done)
+			sd_notify(0, "READY=1");
 		sd_notify(0, "STOPPING=1");
-	else if (new_state == DAEMON_IDLE && old_state == DAEMON_CONFIGURE) {
+	} else if (new_state == DAEMON_IDLE && old_state == DAEMON_CONFIGURE) {
 		sd_notify(0, "READY=1");
 		startup_done = true;
 	} else if (new_state == DAEMON_CONFIGURE && startup_done)
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 07/48] multipathd: improve delayed reconfigure

[mwilck]

From: Martin Wilck <mwilck@suse.com>

When a reconfigure operation is requested, either by the admin
or by some condition multipathd encounters, the current code
attempts to set DAEMON_CONFIGURE state and gives up after a second
if it doesn't succeed. Apart from shutdown, this happens only
if multipathd is either already reconfiguring, or busy in the
path checker loop.

This patch modifies the logic as follows: rather than waiting,
we set a flag that requests a reconfigure operation asap, i.e.
when the current operation is finished and the status switched
to DAEMON_IDLE. In this case, multipathd will not switch to IDLE
but start another reconfigure cycle.

This assumes that if a reconfigure is requested while one is already
running, the admin has made some (additional) changes and wants
multipathd to pull them in. As we can't be sure that the currently
running reconfigure has seen the configuration changes, we need
to start over again.

A positive side effect is less waiting in clients and multipathd.

After this change, the only caller of set_config_state() is
checkerloop(). Waking up every second just to see that DAEMON_RUNNING
couldn't be set makes no sense. Therefore set_config_state() is
changed to wait "forever", or until shutdown is requested. Unless
multipathd completely hangs, the wait will terminate sooner or
later.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli_handlers.c | 10 +----
 multipathd/main.c         | 92 +++++++++++++++++++++++++++++----------
 multipathd/main.h         |  2 +-
 3 files changed, 70 insertions(+), 34 deletions(-)

diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 6d3a0ae..44f76ee 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -1076,17 +1076,9 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
 int
 cli_reconfigure(void * v, char ** reply, int * len, void * data)
 {
-	int rc;
-
 	condlog(2, "reconfigure (operator)");
 
-	rc = set_config_state(DAEMON_CONFIGURE);
-	if (rc == ETIMEDOUT) {
-		condlog(2, "timeout starting reconfiguration");
-		return 1;
-	} else if (rc == EINVAL)
-		/* daemon shutting down */
-		return 1;
+	schedule_reconfigure();
 	return 0;
 }
 
diff --git a/multipathd/main.c b/multipathd/main.c
index 1a1812e..7dc5f49 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -221,6 +221,10 @@ static void do_sd_notify(enum daemon_status old_state,
 	} else if (new_state == DAEMON_CONFIGURE && startup_done)
 		sd_notify(0, "RELOADING=1");
 }
+#else
+static void do_sd_notify(__attribute__((unused)) enum daemon_status old_state,
+			 __attribute__((unused)) enum daemon_status new_state)
+{}
 #endif
 
 static void config_cleanup(__attribute__((unused)) void *arg)
@@ -266,19 +270,38 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 	return st;
 }
 
+/* Don't access this variable without holding config_lock */
+static bool reconfigure_pending;
+
 /* must be called with config_lock held */
 static void __post_config_state(enum daemon_status state)
 {
 	if (state != running_state && running_state != DAEMON_SHUTDOWN) {
-#ifdef USE_SYSTEMD
 		enum daemon_status old_state = running_state;
-#endif
 
+		/*
+		 * Handle a pending reconfigure request.
+		 * DAEMON_IDLE is set from child() after reconfigure(),
+		 * or from checkerloop() after completing checkers.
+		 * In either case, child() will see DAEMON_CONFIGURE
+		 * again and start another reconfigure cycle.
+		 */
+		if (reconfigure_pending && state == DAEMON_IDLE &&
+		    (old_state == DAEMON_CONFIGURE ||
+		     old_state == DAEMON_RUNNING)) {
+			/*
+			 * notify systemd of transient idle state, lest systemd
+			 * thinks the reload lasts forever.
+			 */
+			do_sd_notify(old_state, DAEMON_IDLE);
+			old_state = DAEMON_IDLE;
+			state = DAEMON_CONFIGURE;
+		}
+		if (reconfigure_pending && state == DAEMON_CONFIGURE)
+			reconfigure_pending = false;
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
-#ifdef USE_SYSTEMD
 		do_sd_notify(old_state, state);
-#endif
 	}
 }
 
@@ -290,24 +313,48 @@ void post_config_state(enum daemon_status state)
 	pthread_cleanup_pop(1);
 }
 
-int set_config_state(enum daemon_status state)
+void schedule_reconfigure(void)
+{
+	pthread_mutex_lock(&config_lock);
+	pthread_cleanup_push(config_cleanup, NULL);
+	switch (running_state)
+	{
+	case DAEMON_SHUTDOWN:
+		break;
+	case DAEMON_IDLE:
+		__post_config_state(DAEMON_CONFIGURE);
+		break;
+	case DAEMON_CONFIGURE:
+	case DAEMON_RUNNING:
+		reconfigure_pending = true;
+		break;
+	default:
+		break;
+	}
+	pthread_cleanup_pop(1);
+}
+
+static enum daemon_status set_config_state(enum daemon_status state)
 {
 	int rc = 0;
+	enum daemon_status st;
 
 	pthread_cleanup_push(config_cleanup, NULL);
 	pthread_mutex_lock(&config_lock);
-	if (running_state != state) {
 
-		if (running_state == DAEMON_SHUTDOWN)
-			rc = EINVAL;
-		else
-			rc = __wait_for_state_change(
-				running_state != DAEMON_IDLE, 1000);
-		if (!rc)
-			__post_config_state(state);
+	while (rc == 0 &&
+	       running_state != state &&
+	       running_state != DAEMON_SHUTDOWN &&
+	       running_state != DAEMON_IDLE) {
+		rc = pthread_cond_wait(&config_cond, &config_lock);
 	}
+
+	if (rc == 0 && running_state == DAEMON_IDLE && state != DAEMON_IDLE)
+		__post_config_state(state);
+	st = running_state;
+
 	pthread_cleanup_pop(1);
-	return rc;
+	return st;
 }
 
 struct config *get_multipath_config(void)
@@ -745,7 +792,7 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 			if (delayed_reconfig &&
 			    !need_to_delay_reconfig(vecs)) {
 				condlog(2, "reconfigure (delayed)");
-				set_config_state(DAEMON_CONFIGURE);
+				schedule_reconfigure();
 				return 0;
 			}
 		}
@@ -1856,7 +1903,7 @@ missing_uev_wait_tick(struct vectors *vecs)
 	if (timed_out && delayed_reconfig &&
 	    !need_to_delay_reconfig(vecs)) {
 		condlog(2, "reconfigure (delayed)");
-		set_config_state(DAEMON_CONFIGURE);
+		schedule_reconfigure();
 	}
 }
 
@@ -2495,6 +2542,10 @@ checkerloop (void *ap)
 		int num_paths = 0, strict_timing, rc = 0;
 		unsigned int ticks = 0;
 
+		if (set_config_state(DAEMON_RUNNING) != DAEMON_RUNNING)
+			/* daemon shutdown */
+			break;
+
 		get_monotonic_time(&start_time);
 		if (start_time.tv_sec && last_time.tv_sec) {
 			timespecsub(&start_time, &last_time, &diff_time);
@@ -2510,13 +2561,6 @@ checkerloop (void *ap)
 		if (use_watchdog)
 			sd_notify(0, "WATCHDOG=1");
 #endif
-		rc = set_config_state(DAEMON_RUNNING);
-		if (rc == ETIMEDOUT) {
-			condlog(4, "timeout waiting for DAEMON_IDLE");
-			continue;
-		} else if (rc == EINVAL)
-			/* daemon shutdown */
-			break;
 
 		pthread_cleanup_push(cleanup_lock, &vecs->lock);
 		lock(&vecs->lock);
@@ -2844,7 +2888,7 @@ handle_signals(bool nonfatal)
 		return;
 	if (reconfig_sig) {
 		condlog(2, "reconfigure (signal)");
-		set_config_state(DAEMON_CONFIGURE);
+		schedule_reconfigure();
 	}
 	if (log_reset_sig) {
 		condlog(2, "reset log (signal)");
diff --git a/multipathd/main.h b/multipathd/main.h
index bc1f938..2960a4d 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -37,6 +37,7 @@ void exit_daemon(void);
 const char * daemon_status(void);
 enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 					    unsigned long ms);
+void schedule_reconfigure(void);
 int need_to_delay_reconfig (struct vectors *);
 int reconfigure (struct vectors *);
 int ev_add_path (struct path *, struct vectors *, int);
@@ -44,7 +45,6 @@ int ev_remove_path (struct path *, struct vectors *, int);
 int ev_add_map (char *, const char *, struct vectors *);
 int ev_remove_map (char *, char *, int, struct vectors *);
 int flush_map(struct multipath *, struct vectors *, int);
-int set_config_state(enum daemon_status);
 void * mpath_alloc_prin_response(int prin_sa);
 int prin_do_scsi_ioctl(char *, int rq_servact, struct prin_resp * resp,
 		       int noisy);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 08/48] multipathd: cli.h: formatting improvements

[mwilck]

From: Martin Wilck <mwilck@suse.com>

No functional changes. Just make the code a little easier to read.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.h | 82 ++++++++++++++++++++++++------------------------
 1 file changed, 41 insertions(+), 41 deletions(-)

diff --git a/multipathd/cli.h b/multipathd/cli.h
index fdfb9ae..1e8948a 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -4,83 +4,83 @@
 #include <stdint.h>
 
 enum {
-	__LIST,
+	__LIST,			/*  0 */
 	__ADD,
 	__DEL,
 	__SWITCH,
 	__SUSPEND,
-	__RESUME,
+	__RESUME,			/*  5 */
 	__REINSTATE,
 	__FAIL,
 	__RESIZE,
 	__RESET,
-	__RELOAD,
+	__RELOAD,			/* 10 */
 	__FORCEQ,
 	__DISABLEQ,
 	__RESTOREQ,
 	__PATHS,
-	__MAPS,
+	__MAPS,			/* 15 */
 	__PATH,
 	__MAP,
 	__GROUP,
 	__RECONFIGURE,
-	__DAEMON,
+	__DAEMON,			/* 20 */
 	__STATUS,
 	__STATS,
 	__TOPOLOGY,
 	__CONFIG,
-	__BLACKLIST,
+	__BLACKLIST,			/* 25 */
 	__DEVICES,
 	__RAW,
 	__WILDCARDS,
 	__QUIT,
-	__SHUTDOWN,
+	__SHUTDOWN,			/* 30 */
 	__GETPRSTATUS,
 	__SETPRSTATUS,
 	__UNSETPRSTATUS,
 	__FMT,
-	__JSON,
+	__JSON,			/* 35 */
 	__GETPRKEY,
 	__SETPRKEY,
 	__UNSETPRKEY,
 	__KEY,
-	__LOCAL,
+	__LOCAL,			/* 40 */
 	__SETMARGINAL,
 	__UNSETMARGINAL,
 };
 
-#define LIST		(1 << __LIST)
-#define ADD		(1 << __ADD)
-#define DEL		(1 << __DEL)
-#define SWITCH		(1 << __SWITCH)
-#define SUSPEND		(1 << __SUSPEND)
-#define RESUME		(1 << __RESUME)
-#define REINSTATE	(1 << __REINSTATE)
-#define FAIL		(1 << __FAIL)
-#define RESIZE		(1 << __RESIZE)
-#define RESET		(1 << __RESET)
-#define RELOAD		(1 << __RELOAD)
-#define FORCEQ		(1 << __FORCEQ)
-#define DISABLEQ	(1 << __DISABLEQ)
-#define RESTOREQ	(1 << __RESTOREQ)
-#define PATHS		(1 << __PATHS)
-#define MAPS		(1 << __MAPS)
-#define PATH		(1 << __PATH)
-#define MAP		(1 << __MAP)
-#define GROUP		(1 << __GROUP)
-#define RECONFIGURE	(1 << __RECONFIGURE)
-#define DAEMON		(1 << __DAEMON)
-#define STATUS		(1 << __STATUS)
-#define STATS		(1 << __STATS)
-#define TOPOLOGY	(1 << __TOPOLOGY)
-#define CONFIG		(1 << __CONFIG)
-#define BLACKLIST	(1 << __BLACKLIST)
-#define DEVICES		(1 << __DEVICES)
-#define RAW		(1 << __RAW)
-#define COUNT		(1 << __COUNT)
-#define WILDCARDS	(1 << __WILDCARDS)
-#define QUIT		(1 << __QUIT)
-#define SHUTDOWN	(1 << __SHUTDOWN)
+#define LIST		(1ULL << __LIST)
+#define ADD		(1ULL << __ADD)
+#define DEL		(1ULL << __DEL)
+#define SWITCH		(1ULL << __SWITCH)
+#define SUSPEND	(1ULL << __SUSPEND)
+#define RESUME		(1ULL << __RESUME)
+#define REINSTATE	(1ULL << __REINSTATE)
+#define FAIL		(1ULL << __FAIL)
+#define RESIZE		(1ULL << __RESIZE)
+#define RESET		(1ULL << __RESET)
+#define RELOAD		(1ULL << __RELOAD)
+#define FORCEQ		(1ULL << __FORCEQ)
+#define DISABLEQ	(1ULL << __DISABLEQ)
+#define RESTOREQ	(1ULL << __RESTOREQ)
+#define PATHS		(1ULL << __PATHS)
+#define MAPS		(1ULL << __MAPS)
+#define PATH		(1ULL << __PATH)
+#define MAP		(1ULL << __MAP)
+#define GROUP		(1ULL << __GROUP)
+#define RECONFIGURE	(1ULL << __RECONFIGURE)
+#define DAEMON		(1ULL << __DAEMON)
+#define STATUS		(1ULL << __STATUS)
+#define STATS		(1ULL << __STATS)
+#define TOPOLOGY	(1ULL << __TOPOLOGY)
+#define CONFIG		(1ULL << __CONFIG)
+#define BLACKLIST	(1ULL << __BLACKLIST)
+#define DEVICES	(1ULL << __DEVICES)
+#define RAW		(1ULL << __RAW)
+#define COUNT		(1ULL << __COUNT)
+#define WILDCARDS	(1ULL << __WILDCARDS)
+#define QUIT		(1ULL << __QUIT)
+#define SHUTDOWN	(1ULL << __SHUTDOWN)
 #define GETPRSTATUS	(1ULL << __GETPRSTATUS)
 #define SETPRSTATUS	(1ULL << __SETPRSTATUS)
 #define UNSETPRSTATUS	(1ULL << __UNSETPRSTATUS)
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 09/48] multipathd: cli_del_map: fix reply for delayed action

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Return code 2 from ev_remove_map means that a delayed remove has
been started, which is not the same as failure.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli_handlers.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 44f76ee..2e4b239 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -783,6 +783,9 @@ cli_del_map (void * v, char ** reply, int * len, void * data)
 		return 1;
 	}
 	rc = ev_remove_map(param, alias, minor, vecs);
+	if (rc == 2)
+		*reply = strdup("delayed");
+
 	FREE(alias);
 	return rc;
 }
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 10/48] multipathd: add prototype for cli_handler functions

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Use a typedef instead of spelling out the function type everywhere.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.c |  6 +++---
 multipathd/cli.h | 10 ++++++----
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 4d6c37c..5213813 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -65,7 +65,7 @@ out:
 }
 
 int
-add_handler (uint64_t fp, int (*fn)(void *, char **, int *, void *))
+add_handler (uint64_t fp, cli_handler *fn)
 {
 	struct handler * h;
 
@@ -100,7 +100,7 @@ find_handler (uint64_t fp)
 }
 
 int
-set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *))
+set_handler_callback (uint64_t fp, cli_handler *fn)
 {
 	struct handler * h = find_handler(fp);
 
@@ -112,7 +112,7 @@ set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *))
 }
 
 int
-set_unlocked_handler_callback (uint64_t fp,int (*fn)(void *, char **, int *, void *))
+set_unlocked_handler_callback (uint64_t fp, cli_handler *fn)
 {
 	struct handler * h = find_handler(fp);
 
diff --git a/multipathd/cli.h b/multipathd/cli.h
index 1e8948a..3dac1b4 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -124,16 +124,18 @@ struct key {
 	int has_param;
 };
 
+typedef int (cli_handler)(void *keywords, char **reply, int *len, void *data);
+
 struct handler {
 	uint64_t fingerprint;
 	int locked;
-	int (*fn)(void *, char **, int *, void *);
+	cli_handler *fn;
 };
 
 int alloc_handlers (void);
-int add_handler (uint64_t fp, int (*fn)(void *, char **, int *, void *));
-int set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
-int set_unlocked_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
+int add_handler (uint64_t fp, cli_handler *fn);
+int set_handler_callback (uint64_t fp, cli_handler *fn);
+int set_unlocked_handler_callback (uint64_t fp, cli_handler *fn);
 int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
 int load_keys (void);
 char * get_keyparam (vector v, uint64_t code);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 11/48] multipathd: make all cli_handlers static

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The cli_handler functions are only called from the handler table and
need not be exported.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli_handlers.c | 214 ++++++++++++++++++++++----------------
 multipathd/cli_handlers.h |  61 ++---------
 multipathd/main.c         |  58 +----------
 3 files changed, 134 insertions(+), 199 deletions(-)

diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 2e4b239..1a9c822 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -39,7 +39,7 @@
 		*(__len) = *(__rep) ? sizeof(string_literal) : 0;	\
 	} while (0)
 
-int
+static int
 show_paths (char ** r, int * len, struct vectors * vecs, char * style,
 	    int pretty)
 {
@@ -70,7 +70,7 @@ show_paths (char ** r, int * len, struct vectors * vecs, char * style,
 	return 0;
 }
 
-int
+static int
 show_path (char ** r, int * len, struct vectors * vecs, struct path *pp,
 	   char * style)
 {
@@ -85,7 +85,7 @@ show_path (char ** r, int * len, struct vectors * vecs, struct path *pp,
 	return 0;
 }
 
-int
+static int
 show_map_topology (char ** r, int * len, struct multipath * mpp,
 		   struct vectors * vecs)
 {
@@ -102,7 +102,7 @@ show_map_topology (char ** r, int * len, struct multipath * mpp,
 	return 0;
 }
 
-int
+static int
 show_maps_topology (char ** r, int * len, struct vectors * vecs)
 {
 	STRBUF_ON_STACK(reply);
@@ -128,7 +128,7 @@ show_maps_topology (char ** r, int * len, struct vectors * vecs)
 	return 0;
 }
 
-int
+static int
 show_maps_json (char ** r, int * len, struct vectors * vecs)
 {
 	STRBUF_ON_STACK(reply);
@@ -149,7 +149,7 @@ show_maps_json (char ** r, int * len, struct vectors * vecs)
 	return 0;
 }
 
-int
+static int
 show_map_json (char ** r, int * len, struct multipath * mpp,
 		   struct vectors * vecs)
 {
@@ -194,7 +194,7 @@ reset_stats(struct multipath * mpp)
 	mpp->stat_map_failures = 0;
 }
 
-int
+static int
 cli_list_config (void * v, char ** reply, int * len, void * data)
 {
 	condlog(3, "list config (operator)");
@@ -207,7 +207,7 @@ static void v_free(void *x)
 	vector_free(x);
 }
 
-int
+static int
 cli_list_config_local (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -223,7 +223,7 @@ cli_list_config_local (void * v, char ** reply, int * len, void * data)
 	return ret;
 }
 
-int
+static int
 cli_list_paths (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -233,7 +233,7 @@ cli_list_paths (void * v, char ** reply, int * len, void * data)
 	return show_paths(reply, len, vecs, PRINT_PATH_CHECKER, 1);
 }
 
-int
+static int
 cli_list_paths_fmt (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -244,7 +244,7 @@ cli_list_paths_fmt (void * v, char ** reply, int * len, void * data)
 	return show_paths(reply, len, vecs, fmt, 1);
 }
 
-int
+static int
 cli_list_paths_raw (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -255,7 +255,7 @@ cli_list_paths_raw (void * v, char ** reply, int * len, void * data)
 	return show_paths(reply, len, vecs, fmt, 0);
 }
 
-int
+static int
 cli_list_path (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -272,7 +272,7 @@ cli_list_path (void * v, char ** reply, int * len, void * data)
 	return show_path(reply, len, vecs, pp, "%o");
 }
 
-int
+static int
 cli_list_map_topology (void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -291,7 +291,7 @@ cli_list_map_topology (void * v, char ** reply, int * len, void * data)
 	return show_map_topology(reply, len, mpp, vecs);
 }
 
-int
+static int
 cli_list_maps_topology (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -301,7 +301,7 @@ cli_list_maps_topology (void * v, char ** reply, int * len, void * data)
 	return show_maps_topology(reply, len, vecs);
 }
 
-int
+static int
 cli_list_map_json (void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -320,7 +320,7 @@ cli_list_map_json (void * v, char ** reply, int * len, void * data)
 	return show_map_json(reply, len, mpp, vecs);
 }
 
-int
+static int
 cli_list_maps_json (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -330,7 +330,7 @@ cli_list_maps_json (void * v, char ** reply, int * len, void * data)
 	return show_maps_json(reply, len, vecs);
 }
 
-int
+static int
 cli_list_wildcards (void * v, char ** reply, int * len, void * data)
 {
 	STRBUF_ON_STACK(buf);
@@ -343,7 +343,7 @@ cli_list_wildcards (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 show_status (char ** r, int *len, struct vectors * vecs)
 {
 	STRBUF_ON_STACK(reply);
@@ -356,7 +356,7 @@ show_status (char ** r, int *len, struct vectors * vecs)
 	return 0;
 }
 
-int
+static int
 show_daemon (char ** r, int *len)
 {
 	STRBUF_ON_STACK(reply);
@@ -370,7 +370,7 @@ show_daemon (char ** r, int *len)
 	return 0;
 }
 
-int
+static int
 show_map (char ** r, int *len, struct multipath * mpp, char * style,
 	  int pretty)
 {
@@ -384,7 +384,7 @@ show_map (char ** r, int *len, struct multipath * mpp, char * style,
 	return 0;
 }
 
-int
+static int
 show_maps (char ** r, int *len, struct vectors * vecs, char * style,
 	   int pretty)
 {
@@ -419,7 +419,7 @@ show_maps (char ** r, int *len, struct vectors * vecs, char * style,
 	return 0;
 }
 
-int
+static int
 cli_list_maps_fmt (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -430,7 +430,7 @@ cli_list_maps_fmt (void * v, char ** reply, int * len, void * data)
 	return show_maps(reply, len, vecs, fmt, 1);
 }
 
-int
+static int
 cli_list_maps_raw (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -441,7 +441,7 @@ cli_list_maps_raw (void * v, char ** reply, int * len, void * data)
 	return show_maps(reply, len, vecs, fmt, 0);
 }
 
-int
+static int
 cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -461,27 +461,7 @@ cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
 	return show_map(reply, len, mpp, fmt, 1);
 }
 
-int
-cli_list_map_raw (void * v, char ** reply, int * len, void * data)
-{
-	struct multipath * mpp;
-	struct vectors * vecs = (struct vectors *)data;
-	char * param = get_keyparam(v, MAP);
-	char * fmt = get_keyparam(v, FMT);
-
-	param = convert_dev(param, 0);
-	get_path_layout(vecs->pathvec, 0);
-	get_multipath_layout(vecs->mpvec, 1);
-	mpp = find_mp_by_str(vecs->mpvec, param);
-	if (!mpp)
-		return 1;
-
-	condlog(3, "list map %s fmt %s (operator)", param, fmt);
-
-	return show_map(reply, len, mpp, fmt, 0);
-}
-
-int
+static int
 cli_list_maps (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -491,7 +471,7 @@ cli_list_maps (void * v, char ** reply, int * len, void * data)
 	return show_maps(reply, len, vecs, PRINT_MAP_NAMES, 1);
 }
 
-int
+static int
 cli_list_status (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -501,7 +481,7 @@ cli_list_status (void * v, char ** reply, int * len, void * data)
 	return show_status(reply, len, vecs);
 }
 
-int
+static int
 cli_list_maps_status (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -511,7 +491,7 @@ cli_list_maps_status (void * v, char ** reply, int * len, void * data)
 	return show_maps(reply, len, vecs, PRINT_MAP_STATUS, 1);
 }
 
-int
+static int
 cli_list_maps_stats (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -521,7 +501,7 @@ cli_list_maps_stats (void * v, char ** reply, int * len, void * data)
 	return show_maps(reply, len, vecs, PRINT_MAP_STATS, 1);
 }
 
-int
+static int
 cli_list_daemon (void * v, char ** reply, int * len, void * data)
 {
 	condlog(3, "list daemon (operator)");
@@ -529,7 +509,7 @@ cli_list_daemon (void * v, char ** reply, int * len, void * data)
 	return show_daemon(reply, len);
 }
 
-int
+static int
 cli_reset_maps_stats (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -544,7 +524,7 @@ cli_reset_maps_stats (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_reset_map_stats (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -562,7 +542,7 @@ cli_reset_map_stats (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_add_path (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -678,7 +658,7 @@ blacklisted:
 	return 0;
 }
 
-int
+static int
 cli_del_path (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -701,7 +681,7 @@ cli_del_path (void * v, char ** reply, int * len, void * data)
 	return (ret == REMOVE_PATH_FAILURE);
 }
 
-int
+static int
 cli_add_map (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -761,7 +741,7 @@ cli_add_map (void * v, char ** reply, int * len, void * data)
 	return rc;
 }
 
-int
+static int
 cli_del_map (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -790,7 +770,7 @@ cli_del_map (void * v, char ** reply, int * len, void * data)
 	return rc;
 }
 
-int
+static int
 cli_del_maps (void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -809,7 +789,7 @@ cli_del_maps (void *v, char **reply, int *len, void *data)
 	return ret;
 }
 
-int
+static int
 cli_reload(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -837,7 +817,7 @@ cli_reload(void *v, char **reply, int *len, void *data)
 	return reload_and_sync_map(mpp, vecs, 0);
 }
 
-int resize_map(struct multipath *mpp, unsigned long long size,
+static int resize_map(struct multipath *mpp, unsigned long long size,
 	       struct vectors * vecs)
 {
 	char *params __attribute__((cleanup(cleanup_charp))) = NULL;
@@ -862,7 +842,7 @@ int resize_map(struct multipath *mpp, unsigned long long size,
 	return 0;
 }
 
-int
+static int
 cli_resize(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -927,7 +907,7 @@ cli_resize(void *v, char **reply, int *len, void *data)
 	return 0;
 }
 
-int
+static int
 cli_force_no_daemon_q(void * v, char ** reply, int * len, void * data)
 {
 	struct config *conf;
@@ -940,7 +920,7 @@ cli_force_no_daemon_q(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_restore_no_daemon_q(void * v, char ** reply, int * len, void * data)
 {
 	struct config *conf;
@@ -953,7 +933,7 @@ cli_restore_no_daemon_q(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_restore_queueing(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -994,7 +974,7 @@ cli_restore_queueing(void *v, char **reply, int *len, void *data)
 	return 0;
 }
 
-int
+static int
 cli_restore_all_queueing(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1016,7 +996,7 @@ cli_restore_all_queueing(void *v, char **reply, int *len, void *data)
 	return 0;
 }
 
-int
+static int
 cli_disable_queueing(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1045,7 +1025,7 @@ cli_disable_queueing(void *v, char **reply, int *len, void *data)
 	return 0;
 }
 
-int
+static int
 cli_disable_all_queueing(void *v, char **reply, int *len, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1064,7 +1044,7 @@ cli_disable_all_queueing(void *v, char **reply, int *len, void *data)
 	return 0;
 }
 
-int
+static int
 cli_switch_group(void * v, char ** reply, int * len, void * data)
 {
 	char * mapname = get_keyparam(v, MAP);
@@ -1076,7 +1056,7 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
 	return dm_switchgroup(mapname, groupnum);
 }
 
-int
+static int
 cli_reconfigure(void * v, char ** reply, int * len, void * data)
 {
 	condlog(2, "reconfigure (operator)");
@@ -1085,7 +1065,7 @@ cli_reconfigure(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_suspend(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1115,7 +1095,7 @@ cli_suspend(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_resume(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1147,7 +1127,7 @@ cli_resume(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_reinstate(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1170,7 +1150,7 @@ cli_reinstate(void * v, char ** reply, int * len, void * data)
 	return dm_reinstate_path(pp->mpp->alias, pp->dev_t);
 }
 
-int
+static int
 cli_reassign (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1194,7 +1174,7 @@ cli_reassign (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_fail(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1223,7 +1203,7 @@ cli_fail(void * v, char ** reply, int * len, void * data)
 	return r;
 }
 
-int
+static int
 show_blacklist (char ** r, int * len)
 {
 	STRBUF_ON_STACK(reply);
@@ -1243,7 +1223,7 @@ show_blacklist (char ** r, int * len)
 	return 0;
 }
 
-int
+static int
 cli_list_blacklist (void * v, char ** reply, int * len, void * data)
 {
 	condlog(3, "list blacklist (operator)");
@@ -1251,7 +1231,7 @@ cli_list_blacklist (void * v, char ** reply, int * len, void * data)
 	return show_blacklist(reply, len);
 }
 
-int
+static int
 show_devices (char ** r, int * len, struct vectors *vecs)
 {
 	STRBUF_ON_STACK(reply);
@@ -1272,7 +1252,7 @@ show_devices (char ** r, int * len, struct vectors *vecs)
 	return 0;
 }
 
-int
+static int
 cli_list_devices (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
@@ -1282,13 +1262,13 @@ cli_list_devices (void * v, char ** reply, int * len, void * data)
 	return show_devices(reply, len, vecs);
 }
 
-int
+static int
 cli_quit (void * v, char ** reply, int * len, void * data)
 {
 	return 0;
 }
 
-int
+static int
 cli_shutdown (void * v, char ** reply, int * len, void * data)
 {
 	condlog(3, "shutdown (operator)");
@@ -1296,7 +1276,7 @@ cli_shutdown (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_getprstatus (void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1321,7 +1301,7 @@ cli_getprstatus (void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_setprstatus(void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1344,7 +1324,7 @@ cli_setprstatus(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_unsetprstatus(void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1366,7 +1346,7 @@ cli_unsetprstatus(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_getprkey(void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1398,7 +1378,7 @@ cli_getprkey(void * v, char ** reply, int * len, void * data)
 	return 0;
 }
 
-int
+static int
 cli_unsetprkey(void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1422,7 +1402,7 @@ cli_unsetprkey(void * v, char ** reply, int * len, void * data)
 	return ret;
 }
 
-int
+static int
 cli_setprkey(void * v, char ** reply, int * len, void * data)
 {
 	struct multipath * mpp;
@@ -1454,7 +1434,7 @@ cli_setprkey(void * v, char ** reply, int * len, void * data)
 	return ret;
 }
 
-int cli_set_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_set_marginal(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1481,7 +1461,7 @@ int cli_set_marginal(void * v, char ** reply, int * len, void * data)
 	return reload_and_sync_map(pp->mpp, vecs, 0);
 }
 
-int cli_unset_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_unset_marginal(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1508,7 +1488,7 @@ int cli_unset_marginal(void * v, char ** reply, int * len, void * data)
 	return reload_and_sync_map(pp->mpp, vecs, 0);
 }
 
-int cli_unset_all_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_unset_all_marginal(void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
@@ -1544,3 +1524,63 @@ int cli_unset_all_marginal(void * v, char ** reply, int * len, void * data)
 
 	return reload_and_sync_map(mpp, vecs, 0);
 }
+
+void init_handler_callbacks(void)
+{
+	set_handler_callback(LIST+PATHS, cli_list_paths);
+	set_handler_callback(LIST+PATHS+FMT, cli_list_paths_fmt);
+	set_handler_callback(LIST+PATHS+RAW+FMT, cli_list_paths_raw);
+	set_handler_callback(LIST+PATH, cli_list_path);
+	set_handler_callback(LIST+MAPS, cli_list_maps);
+	set_handler_callback(LIST+STATUS, cli_list_status);
+	set_unlocked_handler_callback(LIST+DAEMON, cli_list_daemon);
+	set_handler_callback(LIST+MAPS+STATUS, cli_list_maps_status);
+	set_handler_callback(LIST+MAPS+STATS, cli_list_maps_stats);
+	set_handler_callback(LIST+MAPS+FMT, cli_list_maps_fmt);
+	set_handler_callback(LIST+MAPS+RAW+FMT, cli_list_maps_raw);
+	set_handler_callback(LIST+MAPS+TOPOLOGY, cli_list_maps_topology);
+	set_handler_callback(LIST+TOPOLOGY, cli_list_maps_topology);
+	set_handler_callback(LIST+MAPS+JSON, cli_list_maps_json);
+	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
+	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
+	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
+	set_handler_callback(LIST+MAP+JSON, cli_list_map_json);
+	set_handler_callback(LIST+CONFIG+LOCAL, cli_list_config_local);
+	set_handler_callback(LIST+CONFIG, cli_list_config);
+	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
+	set_handler_callback(LIST+DEVICES, cli_list_devices);
+	set_handler_callback(LIST+WILDCARDS, cli_list_wildcards);
+	set_handler_callback(RESET+MAPS+STATS, cli_reset_maps_stats);
+	set_handler_callback(RESET+MAP+STATS, cli_reset_map_stats);
+	set_handler_callback(ADD+PATH, cli_add_path);
+	set_handler_callback(DEL+PATH, cli_del_path);
+	set_handler_callback(ADD+MAP, cli_add_map);
+	set_handler_callback(DEL+MAP, cli_del_map);
+	set_handler_callback(DEL+MAPS, cli_del_maps);
+	set_handler_callback(SWITCH+MAP+GROUP, cli_switch_group);
+	set_unlocked_handler_callback(RECONFIGURE, cli_reconfigure);
+	set_handler_callback(SUSPEND+MAP, cli_suspend);
+	set_handler_callback(RESUME+MAP, cli_resume);
+	set_handler_callback(RESIZE+MAP, cli_resize);
+	set_handler_callback(RELOAD+MAP, cli_reload);
+	set_handler_callback(RESET+MAP, cli_reassign);
+	set_handler_callback(REINSTATE+PATH, cli_reinstate);
+	set_handler_callback(FAIL+PATH, cli_fail);
+	set_handler_callback(DISABLEQ+MAP, cli_disable_queueing);
+	set_handler_callback(RESTOREQ+MAP, cli_restore_queueing);
+	set_handler_callback(DISABLEQ+MAPS, cli_disable_all_queueing);
+	set_handler_callback(RESTOREQ+MAPS, cli_restore_all_queueing);
+	set_unlocked_handler_callback(QUIT, cli_quit);
+	set_unlocked_handler_callback(SHUTDOWN, cli_shutdown);
+	set_handler_callback(GETPRSTATUS+MAP, cli_getprstatus);
+	set_handler_callback(SETPRSTATUS+MAP, cli_setprstatus);
+	set_handler_callback(UNSETPRSTATUS+MAP, cli_unsetprstatus);
+	set_handler_callback(FORCEQ+DAEMON, cli_force_no_daemon_q);
+	set_handler_callback(RESTOREQ+DAEMON, cli_restore_no_daemon_q);
+	set_handler_callback(GETPRKEY+MAP, cli_getprkey);
+	set_handler_callback(SETPRKEY+MAP+KEY, cli_setprkey);
+	set_handler_callback(UNSETPRKEY+MAP, cli_unsetprkey);
+	set_handler_callback(SETMARGINAL+PATH, cli_set_marginal);
+	set_handler_callback(UNSETMARGINAL+PATH, cli_unset_marginal);
+	set_handler_callback(UNSETMARGINAL+MAP, cli_unset_all_marginal);
+}
diff --git a/multipathd/cli_handlers.h b/multipathd/cli_handlers.h
index 6f57b42..7eaf847 100644
--- a/multipathd/cli_handlers.h
+++ b/multipathd/cli_handlers.h
@@ -1,55 +1,6 @@
-int cli_list_paths (void * v, char ** reply, int * len, void * data);
-int cli_list_paths_fmt (void * v, char ** reply, int * len, void * data);
-int cli_list_paths_raw (void * v, char ** reply, int * len, void * data);
-int cli_list_path (void * v, char ** reply, int * len, void * data);
-int cli_list_status (void * v, char ** reply, int * len, void * data);
-int cli_list_daemon (void * v, char ** reply, int * len, void * data);
-int cli_list_maps (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_fmt (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_raw (void * v, char ** reply, int * len, void * data);
-int cli_list_map_fmt (void * v, char ** reply, int * len, void * data);
-int cli_list_map_raw (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_status (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_stats (void * v, char ** reply, int * len, void * data);
-int cli_list_map_topology (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_topology (void * v, char ** reply, int * len, void * data);
-int cli_list_map_json (void * v, char ** reply, int * len, void * data);
-int cli_list_maps_json (void * v, char ** reply, int * len, void * data);
-int cli_list_config (void * v, char ** reply, int * len, void * data);
-int cli_list_config_local (void * v, char ** reply, int * len, void * data);
-int cli_list_blacklist (void * v, char ** reply, int * len, void * data);
-int cli_list_devices (void * v, char ** reply, int * len, void * data);
-int cli_list_wildcards (void * v, char ** reply, int * len, void * data);
-int cli_reset_maps_stats (void * v, char ** reply, int * len, void * data);
-int cli_reset_map_stats (void * v, char ** reply, int * len, void * data);
-int cli_add_path (void * v, char ** reply, int * len, void * data);
-int cli_del_path (void * v, char ** reply, int * len, void * data);
-int cli_add_map (void * v, char ** reply, int * len, void * data);
-int cli_del_map (void * v, char ** reply, int * len, void * data);
-int cli_del_maps (void * v, char ** reply, int * len, void * data);
-int cli_switch_group(void * v, char ** reply, int * len, void * data);
-int cli_reconfigure(void * v, char ** reply, int * len, void * data);
-int cli_resize(void * v, char ** reply, int * len, void * data);
-int cli_reload(void * v, char ** reply, int * len, void * data);
-int cli_disable_queueing(void * v, char ** reply, int * len, void * data);
-int cli_disable_all_queueing(void * v, char ** reply, int * len, void * data);
-int cli_restore_queueing(void * v, char ** reply, int * len, void * data);
-int cli_restore_all_queueing(void * v, char ** reply, int * len, void * data);
-int cli_suspend(void * v, char ** reply, int * len, void * data);
-int cli_resume(void * v, char ** reply, int * len, void * data);
-int cli_reinstate(void * v, char ** reply, int * len, void * data);
-int cli_fail(void * v, char ** reply, int * len, void * data);
-int cli_force_no_daemon_q(void * v, char ** reply, int * len, void * data);
-int cli_restore_no_daemon_q(void * v, char ** reply, int * len, void * data);
-int cli_quit(void * v, char ** reply, int * len, void * data);
-int cli_shutdown(void * v, char ** reply, int * len, void * data);
-int cli_reassign (void * v, char ** reply, int * len, void * data);
-int cli_getprstatus(void * v, char ** reply, int * len, void * data);
-int cli_setprstatus(void * v, char ** reply, int * len, void * data);
-int cli_unsetprstatus(void * v, char ** reply, int * len, void * data);
-int cli_getprkey(void * v, char ** reply, int * len, void * data);
-int cli_setprkey(void * v, char ** reply, int * len, void * data);
-int cli_unsetprkey(void * v, char ** reply, int * len, void * data);
-int cli_set_marginal(void * v, char ** reply, int * len, void * data);
-int cli_unset_marginal(void * v, char ** reply, int * len, void * data);
-int cli_unset_all_marginal(void * v, char ** reply, int * len, void * data);
+#ifndef _CLI_HANDLERS_H
+#define _CLI_HANDLERS_H
+
+void init_handler_callbacks(void);
+
+#endif
diff --git a/multipathd/main.c b/multipathd/main.c
index 7dc5f49..268b940 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1704,63 +1704,7 @@ uxlsnrloop (void * ap)
 	/* Tell main thread that thread has started */
 	post_config_state(DAEMON_CONFIGURE);
 
-	set_handler_callback(LIST+PATHS, cli_list_paths);
-	set_handler_callback(LIST+PATHS+FMT, cli_list_paths_fmt);
-	set_handler_callback(LIST+PATHS+RAW+FMT, cli_list_paths_raw);
-	set_handler_callback(LIST+PATH, cli_list_path);
-	set_handler_callback(LIST+MAPS, cli_list_maps);
-	set_handler_callback(LIST+STATUS, cli_list_status);
-	set_unlocked_handler_callback(LIST+DAEMON, cli_list_daemon);
-	set_handler_callback(LIST+MAPS+STATUS, cli_list_maps_status);
-	set_handler_callback(LIST+MAPS+STATS, cli_list_maps_stats);
-	set_handler_callback(LIST+MAPS+FMT, cli_list_maps_fmt);
-	set_handler_callback(LIST+MAPS+RAW+FMT, cli_list_maps_raw);
-	set_handler_callback(LIST+MAPS+TOPOLOGY, cli_list_maps_topology);
-	set_handler_callback(LIST+TOPOLOGY, cli_list_maps_topology);
-	set_handler_callback(LIST+MAPS+JSON, cli_list_maps_json);
-	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
-	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
-	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
-	set_handler_callback(LIST+MAP+JSON, cli_list_map_json);
-	set_handler_callback(LIST+CONFIG+LOCAL, cli_list_config_local);
-	set_handler_callback(LIST+CONFIG, cli_list_config);
-	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
-	set_handler_callback(LIST+DEVICES, cli_list_devices);
-	set_handler_callback(LIST+WILDCARDS, cli_list_wildcards);
-	set_handler_callback(RESET+MAPS+STATS, cli_reset_maps_stats);
-	set_handler_callback(RESET+MAP+STATS, cli_reset_map_stats);
-	set_handler_callback(ADD+PATH, cli_add_path);
-	set_handler_callback(DEL+PATH, cli_del_path);
-	set_handler_callback(ADD+MAP, cli_add_map);
-	set_handler_callback(DEL+MAP, cli_del_map);
-	set_handler_callback(DEL+MAPS, cli_del_maps);
-	set_handler_callback(SWITCH+MAP+GROUP, cli_switch_group);
-	set_unlocked_handler_callback(RECONFIGURE, cli_reconfigure);
-	set_handler_callback(SUSPEND+MAP, cli_suspend);
-	set_handler_callback(RESUME+MAP, cli_resume);
-	set_handler_callback(RESIZE+MAP, cli_resize);
-	set_handler_callback(RELOAD+MAP, cli_reload);
-	set_handler_callback(RESET+MAP, cli_reassign);
-	set_handler_callback(REINSTATE+PATH, cli_reinstate);
-	set_handler_callback(FAIL+PATH, cli_fail);
-	set_handler_callback(DISABLEQ+MAP, cli_disable_queueing);
-	set_handler_callback(RESTOREQ+MAP, cli_restore_queueing);
-	set_handler_callback(DISABLEQ+MAPS, cli_disable_all_queueing);
-	set_handler_callback(RESTOREQ+MAPS, cli_restore_all_queueing);
-	set_unlocked_handler_callback(QUIT, cli_quit);
-	set_unlocked_handler_callback(SHUTDOWN, cli_shutdown);
-	set_handler_callback(GETPRSTATUS+MAP, cli_getprstatus);
-	set_handler_callback(SETPRSTATUS+MAP, cli_setprstatus);
-	set_handler_callback(UNSETPRSTATUS+MAP, cli_unsetprstatus);
-	set_handler_callback(FORCEQ+DAEMON, cli_force_no_daemon_q);
-	set_handler_callback(RESTOREQ+DAEMON, cli_restore_no_daemon_q);
-	set_handler_callback(GETPRKEY+MAP, cli_getprkey);
-	set_handler_callback(SETPRKEY+MAP+KEY, cli_setprkey);
-	set_handler_callback(UNSETPRKEY+MAP, cli_unsetprkey);
-	set_handler_callback(SETMARGINAL+PATH, cli_set_marginal);
-	set_handler_callback(UNSETMARGINAL+PATH, cli_unset_marginal);
-	set_handler_callback(UNSETMARGINAL+MAP, cli_unset_all_marginal);
-
+	init_handler_callbacks();
 	umask(077);
 	uxsock_listen(&uxsock_trigger, ux_sock, ap);
 
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 12/48] multipathd: add and set cli_handlers in a single step

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Modify set_handler_callback() such that a missing slot is created
if no matching slot is found. This way, we can skip the initialization
with NULL handlers on startup. Assigning the same handler multiple
times would be a bug which is tested with assert().

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.c | 95 ++++++++----------------------------------------
 multipathd/cli.h |  7 ++--
 2 files changed, 19 insertions(+), 83 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 5213813..6e5056e 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -4,6 +4,7 @@
 #include <sys/time.h>
 #include <errno.h>
 #include <pthread.h>
+#include <assert.h>
 #include "memory.h"
 #include "vector.h"
 #include "structs.h"
@@ -64,26 +65,26 @@ out:
 	return 1;
 }
 
-int
-add_handler (uint64_t fp, cli_handler *fn)
+static struct handler *add_handler(uint64_t fp, cli_handler *fn, bool locked)
 {
 	struct handler * h;
 
 	h = alloc_handler();
 
-	if (!h)
-		return 1;
+	if (h == NULL)
+		return NULL;
 
 	if (!vector_alloc_slot(handlers)) {
 		FREE(h);
-		return 1;
+		return NULL;
 	}
 
 	vector_set_slot(handlers, h);
 	h->fingerprint = fp;
 	h->fn = fn;
+	h->locked = locked;
 
-	return 0;
+	return h;
 }
 
 static struct handler *
@@ -100,26 +101,17 @@ find_handler (uint64_t fp)
 }
 
 int
-set_handler_callback (uint64_t fp, cli_handler *fn)
+__set_handler_callback (uint64_t fp, cli_handler *fn, bool locked)
 {
-	struct handler * h = find_handler(fp);
+	struct handler *h;
 
-	if (!h)
+	assert(find_handler(fp) == NULL);
+        h = add_handler(fp, fn, locked);
+        if (!h) {
+		condlog(0, "%s: failed to set handler for code %"PRIu64,
+			__func__, fp);
 		return 1;
-	h->fn = fn;
-	h->locked = 1;
-	return 0;
-}
-
-int
-set_unlocked_handler_callback (uint64_t fp, cli_handler *fn)
-{
-	struct handler * h = find_handler(fp);
-
-	if (!h)
-		return 1;
-	h->fn = fn;
-	h->locked = 0;
+	}
 	return 0;
 }
 
@@ -513,63 +505,6 @@ cli_init (void) {
 	if (alloc_handlers())
 		return 1;
 
-	add_handler(LIST+PATHS, NULL);
-	add_handler(LIST+PATHS+FMT, NULL);
-	add_handler(LIST+PATHS+RAW+FMT, NULL);
-	add_handler(LIST+PATH, NULL);
-	add_handler(LIST+STATUS, NULL);
-	add_handler(LIST+DAEMON, NULL);
-	add_handler(LIST+MAPS, NULL);
-	add_handler(LIST+MAPS+STATUS, NULL);
-	add_handler(LIST+MAPS+STATS, NULL);
-	add_handler(LIST+MAPS+FMT, NULL);
-	add_handler(LIST+MAPS+RAW+FMT, NULL);
-	add_handler(LIST+MAPS+TOPOLOGY, NULL);
-	add_handler(LIST+MAPS+JSON, NULL);
-	add_handler(LIST+TOPOLOGY, NULL);
-	add_handler(LIST+MAP+TOPOLOGY, NULL);
-	add_handler(LIST+MAP+JSON, NULL);
-	add_handler(LIST+MAP+FMT, NULL);
-	add_handler(LIST+MAP+RAW+FMT, NULL);
-	add_handler(LIST+CONFIG, NULL);
-	add_handler(LIST+CONFIG+LOCAL, NULL);
-	add_handler(LIST+BLACKLIST, NULL);
-	add_handler(LIST+DEVICES, NULL);
-	add_handler(LIST+WILDCARDS, NULL);
-	add_handler(RESET+MAPS+STATS, NULL);
-	add_handler(RESET+MAP+STATS, NULL);
-	add_handler(ADD+PATH, NULL);
-	add_handler(DEL+PATH, NULL);
-	add_handler(ADD+MAP, NULL);
-	add_handler(DEL+MAP, NULL);
-	add_handler(DEL+MAPS, NULL);
-	add_handler(SWITCH+MAP+GROUP, NULL);
-	add_handler(RECONFIGURE, NULL);
-	add_handler(SUSPEND+MAP, NULL);
-	add_handler(RESUME+MAP, NULL);
-	add_handler(RESIZE+MAP, NULL);
-	add_handler(RESET+MAP, NULL);
-	add_handler(RELOAD+MAP, NULL);
-	add_handler(DISABLEQ+MAP, NULL);
-	add_handler(RESTOREQ+MAP, NULL);
-	add_handler(DISABLEQ+MAPS, NULL);
-	add_handler(RESTOREQ+MAPS, NULL);
-	add_handler(REINSTATE+PATH, NULL);
-	add_handler(FAIL+PATH, NULL);
-	add_handler(QUIT, NULL);
-	add_handler(SHUTDOWN, NULL);
-	add_handler(GETPRSTATUS+MAP, NULL);
-	add_handler(SETPRSTATUS+MAP, NULL);
-	add_handler(UNSETPRSTATUS+MAP, NULL);
-	add_handler(GETPRKEY+MAP, NULL);
-	add_handler(SETPRKEY+MAP+KEY, NULL);
-	add_handler(UNSETPRKEY+MAP, NULL);
-	add_handler(FORCEQ+DAEMON, NULL);
-	add_handler(RESTOREQ+DAEMON, NULL);
-	add_handler(SETMARGINAL+PATH, NULL);
-	add_handler(UNSETMARGINAL+PATH, NULL);
-	add_handler(UNSETMARGINAL+MAP, NULL);
-
 	return 0;
 }
 
diff --git a/multipathd/cli.h b/multipathd/cli.h
index 3dac1b4..479a745 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -133,9 +133,10 @@ struct handler {
 };
 
 int alloc_handlers (void);
-int add_handler (uint64_t fp, cli_handler *fn);
-int set_handler_callback (uint64_t fp, cli_handler *fn);
-int set_unlocked_handler_callback (uint64_t fp, cli_handler *fn);
+int __set_handler_callback (uint64_t fp, cli_handler *fn, bool locked);
+#define set_handler_callback(fp, fn) __set_handler_callback(fp, fn, true)
+#define set_unlocked_handler_callback(fp, fn) __set_handler_callback(fp, fn, false)
+
 int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
 int load_keys (void);
 char * get_keyparam (vector v, uint64_t code);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 13/48] multipathd: cli.c: use ESRCH for "command not found"

[mwilck]

From: Martin Wilck <mwilck@suse.com>

EAGAIN is too generic, and doesn't fit semantically either.
ESRCH in't used anywhere else in our code.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 6e5056e..93a3f5d 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -251,7 +251,7 @@ find_key (const char * str)
  *
  * returns:
  * ENOMEM: not enough memory to allocate command
- * EAGAIN: command not found
+ * ESRCH: command not found
  * EINVAL: argument missing for command
  */
 static int
@@ -286,7 +286,7 @@ get_cmdvec (char * cmd, vector *v)
 		}
 		kw = find_key(buff);
 		if (!kw) {
-			r = EAGAIN;
+			r = ESRCH;
 			goto out;
 		}
 		cmdkw = alloc_key();
@@ -376,7 +376,7 @@ do_genhelp(struct strbuf *reply, const char *cmd, int error) {
 	case ENOMEM:
 		rc = print_strbuf(reply, "%s: Not enough memory\n", cmd);
 		break;
-	case EAGAIN:
+	case ESRCH:
 		rc = print_strbuf(reply, "%s: not found\n", cmd);
 		break;
 	case EINVAL:
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 14/48] multipathd: uxlsnr: avoid stalled clients during reconfigure

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Since 47cc1d3 ("multipathd: fix client response for socket
activation"), we hold back clients while reconfigure is running.
The idea of 47cc1d3 was to fix the behavior during initial
start up. When multipathd reconfigures itself during runtime,
and the reconfiguration takes a long time (a minute or more is
not unusual in big configurations), clients will time out with
no response ("timeout receiving packet"). Waiting for reconfigure
to finish breaks our timeout handling.

Therefore we should only apply the logic of 47cc1d3 during initial
configuration. In this case, the client that triggered socket
activation may still encounter a timeout, but there's not much we can
do about that.

Fixes: 47cc1d3 ("multipathd: fix client response for socket activation")
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/main.c   |  9 +++++++++
 multipathd/uxlsnr.c | 12 ------------
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 268b940..0820a05 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1706,6 +1706,15 @@ uxlsnrloop (void * ap)
 
 	init_handler_callbacks();
 	umask(077);
+
+	/*
+	 * Wait for initial reconfiguration to finish, while
+	 * hadling signals
+	 */
+	while (wait_for_state_change_if(DAEMON_CONFIGURE, 50)
+	       == DAEMON_CONFIGURE)
+		handle_signals(false);
+
 	uxsock_listen(&uxsock_trigger, ux_sock, ap);
 
 out_sock:
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index dbee0d6..20efbd3 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -391,18 +391,6 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 			continue;
 		}
 
-		/*
-		 * Client connection. We shouldn't answer while we're
-		 * configuring - nothing may be configured yet.
-		 * But we can't wait forever either, because this thread
-		 * must handle signals. So wait a short while only.
-		 */
-		if (wait_for_state_change_if(DAEMON_CONFIGURE, 10)
-		    == DAEMON_CONFIGURE) {
-			handle_signals(false);
-			continue;
-		}
-
 		/* see if a client wants to speak to us */
 		for (i = POLLFDS_BASE; i < n_pfds; i++) {
 			if (polls[i].revents & POLLIN) {
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 15/48] multipathd: uxlsnr: handle client HUP

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The unix socket listener thread doesn't even look at the revents
returned by poll() while the daemon is configuring. This may cause a
closed client socket to be kept open for a long time by the server,
while the listener basically performs a busy loop, as ppoll() always
returns immediately as long as the POLLHUP condition exists.

Worse, it can happen that multipathd reads data from such a closed
client socket after the client has disconnected. See the description
of POLLHUP in poll(2).

Close connections immediately if HUP is received.

Also, use the fd in log messages to identify the client rather
than the random index.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 20efbd3..7e88538 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -393,7 +393,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 
 		/* see if a client wants to speak to us */
 		for (i = POLLFDS_BASE; i < n_pfds; i++) {
-			if (polls[i].revents & POLLIN) {
+			if (polls[i].revents & (POLLIN|POLLHUP|POLLERR)) {
 				struct timespec start_time;
 
 				c = NULL;
@@ -410,6 +410,12 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 						i, polls[i].fd);
 					continue;
 				}
+				if (polls[i].revents & (POLLHUP|POLLERR)) {
+					condlog(4, "cli[%d]: Disconnected",
+						c->fd);
+					dead_client(c);
+					continue;
+				}
 				get_monotonic_time(&start_time);
 				if (recv_packet_from_client(c->fd, &inbuf,
 							    uxsock_timeout)
@@ -423,7 +429,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 					continue;
 				}
 				condlog(4, "cli[%d]: Got request [%s]",
-					i, inbuf);
+					polls[i].fd, inbuf);
 				uxsock_trigger(inbuf, &reply, &rlen,
 					       _socket_client_is_root(c->fd),
 					       trigger_data);
@@ -434,7 +440,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 					} else {
 						condlog(4, "cli[%d]: "
 							"Reply [%d bytes]",
-							i, rlen);
+							polls[i].fd, rlen);
 					}
 					FREE(reply);
 					reply = NULL;
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 16/48] multipathd: uxlsnr: use symbolic values for pollfd indices

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Avoid hardcoding the indices as 0, 1, 2...

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 7e88538..6506109 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -46,8 +46,13 @@ struct client {
 	int fd;
 };
 
-/* The number of fds we poll on, other than individual client connections */
-#define POLLFDS_BASE 2
+/* Indices for array of poll fds */
+enum {
+	POLLFD_UX = 0,
+	POLLFD_NOTIFY,
+	POLLFDS_BASE,
+};
+
 #define POLLFD_CHUNK (4096 / sizeof(struct pollfd))
 /* Minimum mumber of pollfds to reserve for clients */
 #define MIN_POLLS (POLLFD_CHUNK - POLLFDS_BASE)
@@ -339,8 +344,8 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 			}
 		}
 		if (num_clients < MAX_CLIENTS) {
-			polls[0].fd = ux_sock;
-			polls[0].events = POLLIN;
+			polls[POLLFD_UX].fd = ux_sock;
+			polls[POLLFD_UX].events = POLLIN;
 		} else {
 			/*
 			 * New clients can't connect, num_clients won't grow
@@ -348,15 +353,15 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 			 */
 			condlog(1, "%s: max client connections reached, pausing polling",
 				__func__);
-			polls[0].fd = -1;
+			polls[POLLFD_UX].fd = -1;
 		}
 
 		reset_watch(notify_fd, &wds, &sequence_nr);
 		if (notify_fd == -1 || (wds.conf_wd == -1 && wds.dir_wd == -1))
-			polls[1].fd = -1;
+			polls[POLLFD_NOTIFY].fd = -1;
 		else
-			polls[1].fd = notify_fd;
-		polls[1].events = POLLIN;
+			polls[POLLFD_NOTIFY].fd = notify_fd;
+		polls[POLLFD_NOTIFY].events = POLLIN;
 
 		/* setup the clients */
 		i = POLLFDS_BASE;
@@ -454,12 +459,12 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 		handle_signals(true);
 
 		/* see if we got a new client */
-		if (polls[0].revents & POLLIN) {
+		if (polls[POLLFD_UX].revents & POLLIN) {
 			new_client(ux_sock);
 		}
 
 		/* handle inotify events on config files */
-		if (polls[1].revents & POLLIN)
+		if (polls[POLLFD_NOTIFY].revents & POLLIN)
 			handle_inotify(notify_fd, &wds);
 	}
 
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 17/48] multipathd: uxlsnr: avoid using fd -1 in ppoll()

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Minor edit: if notifications are off, we set the poll fd to
-1 but still use the POLLIN mask. It looks nicer if to poll
the correct fd, but reset the event mask to 0 if we're not
actually interested in it.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 6506109..98a9f71 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -357,11 +357,11 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
 		}
 
 		reset_watch(notify_fd, &wds, &sequence_nr);
+		polls[POLLFD_NOTIFY].fd = notify_fd;
 		if (notify_fd == -1 || (wds.conf_wd == -1 && wds.dir_wd == -1))
-			polls[POLLFD_NOTIFY].fd = -1;
+			polls[POLLFD_NOTIFY].events = 0;
 		else
-			polls[POLLFD_NOTIFY].fd = notify_fd;
-		polls[POLLFD_NOTIFY].events = POLLIN;
+			polls[POLLFD_NOTIFY].events = POLLIN;
 
 		/* setup the clients */
 		i = POLLFDS_BASE;
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 18/48] multipathd: uxlsnr: data structure for stateful client connection

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Currently the uxlsnr handles each client request (receive requset -
handle request - respond) in a single loop iteration. This has
severe disadvantages. In particular, the code may wait in poll()
called from read_all(), or wait for the vecs lock, while other
clients are ready to be serviced or signals to be handled.

This patch adds some fields to "struct client" which will be used
by later patches to change this into a state machine that basically
waits only in place, the ppoll() call in uxsock_listen().

For now, we just introduce and initialize the fields.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 98a9f71..e701a1c 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -40,10 +40,30 @@
 #include "main.h"
 #include "cli.h"
 #include "uxlsnr.h"
+#include "strbuf.h"
+
+/* state of client connection */
+enum {
+	CLT_RECV,
+	CLT_PARSE,
+	CLT_WAIT_LOCK,
+	CLT_WORK,
+	CLT_SEND,
+};
 
 struct client {
 	struct list_head node;
+	struct timespec expires;
+	int state;
 	int fd;
+	vector cmdvec;
+	/* NUL byte at end */
+	char cmd[_MAX_CMD_LEN + 1];
+	struct strbuf reply;
+	struct handler *handler;
+	size_t cmd_len, len;
+	int error;
+	bool is_root;
 };
 
 /* Indices for array of poll fds */
@@ -104,14 +124,14 @@ static void new_client(int ux_sock)
 	if (fd == -1)
 		return;
 
-	c = (struct client *)MALLOC(sizeof(*c));
+	c = calloc(1, sizeof(*c));
 	if (!c) {
 		close(fd);
 		return;
 	}
-	memset(c, 0, sizeof(*c));
 	INIT_LIST_HEAD(&c->node);
 	c->fd = fd;
+	c->state = CLT_RECV;
 
 	/* put it in our linked list */
 	pthread_mutex_lock(&client_lock);
@@ -127,6 +147,9 @@ static void _dead_client(struct client *c)
 	int fd = c->fd;
 	list_del_init(&c->node);
 	c->fd = -1;
+	reset_strbuf(&c->reply);
+	if (c->cmdvec)
+		free_keys(c->cmdvec);
 	FREE(c);
 	close(fd);
 }
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 19/48] multipathd: move uxsock_trigger() to uxlsnr.c

[mwilck]

From: Martin Wilck <mwilck@suse.com>

uxsock_trigger() really belongs into cli.c. I suppose that way back in
the past there were strong reasons to call this function via a
pointer. I don't think these reasons are valid any more. Moving
the function to cli.c allows restructuring the code.

No functional changes.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/main.c   | 44 +-------------------------------------------
 multipathd/uxlsnr.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
 multipathd/uxlsnr.h |  4 +---
 3 files changed, 44 insertions(+), 48 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 0820a05..dbe6dc7 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1537,48 +1537,6 @@ map_discovery (struct vectors * vecs)
 	return 0;
 }
 
-int
-uxsock_trigger (char * str, char ** reply, int * len, bool is_root,
-		void * trigger_data)
-{
-	struct vectors * vecs;
-	int r;
-
-	*reply = NULL;
-	*len = 0;
-	vecs = (struct vectors *)trigger_data;
-
-	if ((str != NULL) && (is_root == false) &&
-	    (strncmp(str, "list", strlen("list")) != 0) &&
-	    (strncmp(str, "show", strlen("show")) != 0)) {
-		*reply = STRDUP("permission deny: need to be root");
-		if (*reply)
-			*len = strlen(*reply) + 1;
-		return 1;
-	}
-
-	r = parse_cmd(str, reply, len, vecs, uxsock_timeout / 1000);
-
-	if (r > 0) {
-		if (r == ETIMEDOUT)
-			*reply = STRDUP("timeout\n");
-		else
-			*reply = STRDUP("fail\n");
-		if (*reply)
-			*len = strlen(*reply) + 1;
-		r = 1;
-	}
-	else if (!r && *len == 0) {
-		*reply = STRDUP("ok\n");
-		if (*reply)
-			*len = strlen(*reply) + 1;
-		r = 0;
-	}
-	/* else if (r < 0) leave *reply alone */
-
-	return r;
-}
-
 int
 uev_trigger (struct uevent * uev, void * trigger_data)
 {
@@ -1715,7 +1673,7 @@ uxlsnrloop (void * ap)
 	       == DAEMON_CONFIGURE)
 		handle_signals(false);
 
-	uxsock_listen(&uxsock_trigger, ux_sock, ap);
+	uxsock_listen(ux_sock, ap);
 
 out_sock:
 	pthread_cleanup_pop(1); /* uxsock_cleanup */
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index e701a1c..622aac1 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -311,11 +311,51 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 		condlog(1, "Multipath configuration updated.\nReload multipathd for changes to take effect");
 }
 
+static int uxsock_trigger(char *str, char **reply, int *len,
+			  bool is_root, void *trigger_data)
+{
+	struct vectors * vecs;
+	int r;
+
+	*reply = NULL;
+	*len = 0;
+	vecs = (struct vectors *)trigger_data;
+
+	if ((str != NULL) && (is_root == false) &&
+	    (strncmp(str, "list", strlen("list")) != 0) &&
+	    (strncmp(str, "show", strlen("show")) != 0)) {
+		*reply = STRDUP("permission deny: need to be root");
+		if (*reply)
+			*len = strlen(*reply) + 1;
+		return 1;
+	}
+
+	r = parse_cmd(str, reply, len, vecs, uxsock_timeout / 1000);
+
+	if (r > 0) {
+		if (r == ETIMEDOUT)
+			*reply = STRDUP("timeout\n");
+		else
+			*reply = STRDUP("fail\n");
+		if (*reply)
+			*len = strlen(*reply) + 1;
+		r = 1;
+	}
+	else if (!r && *len == 0) {
+		*reply = STRDUP("ok\n");
+		if (*reply)
+			*len = strlen(*reply) + 1;
+		r = 0;
+	}
+	/* else if (r < 0) leave *reply alone */
+
+	return r;
+}
+
 /*
  * entry point
  */
-void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
-		     void * trigger_data)
+void *uxsock_listen(long ux_sock, void *trigger_data)
 {
 	int rlen;
 	char *inbuf;
diff --git a/multipathd/uxlsnr.h b/multipathd/uxlsnr.h
index 18f008d..60c3a2c 100644
--- a/multipathd/uxlsnr.h
+++ b/multipathd/uxlsnr.h
@@ -3,10 +3,8 @@
 
 #include <stdbool.h>
 
-typedef int (uxsock_trigger_fn)(char *, char **, int *, bool, void *);
-
 void uxsock_cleanup(void *arg);
-void *uxsock_listen(uxsock_trigger_fn uxsock_trigger, long ux_sock,
+void *uxsock_listen(long ux_sock,
 		    void * trigger_data);
 
 #endif
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 20/48] multipathd: move parse_cmd() to uxlsnr.c

[mwilck]

From: Martin Wilck <mwilck@suse.com>

parse_cmd() does more than the name says - it parses, executes
handlers, and even provides reply strings for some cases. This doesn't
work well with the state machine idea. Thus move it to uxlsnr.c,
where later patches will move some functionality elsewhere.

No functional changes.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.c    | 74 +++++----------------------------------------
 multipathd/cli.h    |  5 ++-
 multipathd/uxlsnr.c | 61 +++++++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 67 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 93a3f5d..eb94c75 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -254,8 +254,7 @@ find_key (const char * str)
  * ESRCH: command not found
  * EINVAL: argument missing for command
  */
-static int
-get_cmdvec (char * cmd, vector *v)
+int get_cmdvec (char *cmd, vector *v)
 {
 	int i;
 	int r = 0;
@@ -320,7 +319,7 @@ out:
 }
 
 static uint64_t
-fingerprint(vector vec)
+fingerprint(const struct _vector *vec)
 {
 	int i;
 	uint64_t fp = 0;
@@ -335,6 +334,11 @@ fingerprint(vector vec)
 	return fp;
 }
 
+struct handler *find_handler_for_cmdvec(const struct _vector *v)
+{
+	return find_handler(fingerprint(v));
+}
+
 int
 alloc_handlers (void)
 {
@@ -413,8 +417,7 @@ do_genhelp(struct strbuf *reply, const char *cmd, int error) {
 }
 
 
-static char *
-genhelp_handler (const char *cmd, int error)
+char *genhelp_handler(const char *cmd, int error)
 {
 	STRBUF_ON_STACK(reply);
 
@@ -423,67 +426,6 @@ genhelp_handler (const char *cmd, int error)
 	return steal_strbuf_str(&reply);
 }
 
-int
-parse_cmd (char * cmd, char ** reply, int * len, void * data, int timeout )
-{
-	int r;
-	struct handler * h;
-	vector cmdvec = NULL;
-	struct timespec tmo;
-
-	r = get_cmdvec(cmd, &cmdvec);
-
-	if (r) {
-		*reply = genhelp_handler(cmd, r);
-		if (*reply == NULL)
-			return EINVAL;
-		*len = strlen(*reply) + 1;
-		return 0;
-	}
-
-	h = find_handler(fingerprint(cmdvec));
-
-	if (!h || !h->fn) {
-		free_keys(cmdvec);
-		*reply = genhelp_handler(cmd, EINVAL);
-		if (*reply == NULL)
-			return EINVAL;
-		*len = strlen(*reply) + 1;
-		return 0;
-	}
-
-	/*
-	 * execute handler
-	 */
-	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
-		tmo.tv_sec += timeout;
-	} else {
-		tmo.tv_sec = 0;
-	}
-	if (h->locked) {
-		int locked = 0;
-		struct vectors * vecs = (struct vectors *)data;
-
-		pthread_cleanup_push(cleanup_lock, &vecs->lock);
-		if (tmo.tv_sec) {
-			r = timedlock(&vecs->lock, &tmo);
-		} else {
-			lock(&vecs->lock);
-			r = 0;
-		}
-		if (r == 0) {
-			locked = 1;
-			pthread_testcancel();
-			r = h->fn(cmdvec, reply, len, data);
-		}
-		pthread_cleanup_pop(locked);
-	} else
-		r = h->fn(cmdvec, reply, len, data);
-	free_keys(cmdvec);
-
-	return r;
-}
-
 char *
 get_keyparam (vector v, uint64_t code)
 {
diff --git a/multipathd/cli.h b/multipathd/cli.h
index 479a745..f795826 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -137,7 +137,10 @@ int __set_handler_callback (uint64_t fp, cli_handler *fn, bool locked);
 #define set_handler_callback(fp, fn) __set_handler_callback(fp, fn, true)
 #define set_unlocked_handler_callback(fp, fn) __set_handler_callback(fp, fn, false)
 
-int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
+int get_cmdvec (char *cmd, vector *v);
+struct handler *find_handler_for_cmdvec(const struct _vector *v);
+char *genhelp_handler (const char *cmd, int error);
+
 int load_keys (void);
 char * get_keyparam (vector v, uint64_t code);
 void free_keys (vector vec);
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 622aac1..cbbcb2c 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -311,6 +311,67 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 		condlog(1, "Multipath configuration updated.\nReload multipathd for changes to take effect");
 }
 
+static int parse_cmd (char *cmd, char **reply, int *len, void *data,
+		      int timeout)
+{
+	int r;
+	struct handler * h;
+	vector cmdvec = NULL;
+	struct timespec tmo;
+
+	r = get_cmdvec(cmd, &cmdvec);
+
+	if (r) {
+		*reply = genhelp_handler(cmd, r);
+		if (*reply == NULL)
+			return EINVAL;
+		*len = strlen(*reply) + 1;
+		return 0;
+	}
+
+	h = find_handler_for_cmdvec(cmdvec);
+
+	if (!h || !h->fn) {
+		free_keys(cmdvec);
+		*reply = genhelp_handler(cmd, EINVAL);
+		if (*reply == NULL)
+			return EINVAL;
+		*len = strlen(*reply) + 1;
+		return 0;
+	}
+
+	/*
+	 * execute handler
+	 */
+	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
+		tmo.tv_sec += timeout;
+	} else {
+		tmo.tv_sec = 0;
+	}
+	if (h->locked) {
+		int locked = 0;
+		struct vectors * vecs = (struct vectors *)data;
+
+		pthread_cleanup_push(cleanup_lock, &vecs->lock);
+		if (tmo.tv_sec) {
+			r = timedlock(&vecs->lock, &tmo);
+		} else {
+			lock(&vecs->lock);
+			r = 0;
+		}
+		if (r == 0) {
+			locked = 1;
+			pthread_testcancel();
+			r = h->fn(cmdvec, reply, len, data);
+		}
+		pthread_cleanup_pop(locked);
+	} else
+		r = h->fn(cmdvec, reply, len, data);
+	free_keys(cmdvec);
+
+	return r;
+}
+
 static int uxsock_trigger(char *str, char **reply, int *len,
 			  bool is_root, void *trigger_data)
 {
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 21/48] multipathd: uxlsnr: remove check_timeout()

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This function just prints a warning, anyway. If this warning
is printed, the client will see a timeout and print a warning, too.
A later patch will re-introduce this function with real functionality.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 21 ---------------------
 1 file changed, 21 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index cbbcb2c..147f81a 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -168,25 +168,6 @@ static void free_polls (void)
 		FREE(polls);
 }
 
-static void check_timeout(struct timespec start_time, char *inbuf,
-		   unsigned int timeout)
-{
-	struct timespec diff_time, end_time;
-
-	if (start_time.tv_sec) {
-		unsigned long msecs;
-
-		get_monotonic_time(&end_time);
-		timespecsub(&end_time, &start_time, &diff_time);
-		msecs = diff_time.tv_sec * 1000 +
-			diff_time.tv_nsec / (1000 * 1000);
-		if (msecs > timeout)
-			condlog(2, "cli cmd '%s' timeout reached "
-				"after %ld.%06lu secs", inbuf,
-				(long)diff_time.tv_sec, diff_time.tv_nsec / 1000);
-	}
-}
-
 void uxsock_cleanup(void *arg)
 {
 	struct client *client_loop;
@@ -574,8 +555,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 					FREE(reply);
 					reply = NULL;
 				}
-				check_timeout(start_time, inbuf,
-					      uxsock_timeout);
 				FREE(inbuf);
 			}
 		}
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 22/48] multipathd: uxlsnr: move client handling to separate function

[mwilck]

From: Martin Wilck <mwilck@suse.com>

No functional changes at this point. handle_client() will become
the state machine for handling client requests.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 67 ++++++++++++++++++++++-----------------------
 1 file changed, 32 insertions(+), 35 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 147f81a..2fb23c8 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -394,14 +394,42 @@ static int uxsock_trigger(char *str, char **reply, int *len,
 	return r;
 }
 
+static void handle_client(struct client *c, void *trigger_data)
+{
+	int rlen;
+	char *inbuf, *reply;
+
+	if (recv_packet_from_client(c->fd, &inbuf, uxsock_timeout) != 0) {
+		dead_client(c);
+		return;
+	}
+
+	if (!inbuf) {
+		condlog(4, "recv_packet_from_client get null request");
+		return;
+	}
+
+	condlog(4, "cli[%d]: Got request [%s]", c->fd, inbuf);
+	uxsock_trigger(inbuf, &reply, &rlen,
+		       _socket_client_is_root(c->fd),
+		       trigger_data);
+
+	if (reply) {
+		if (send_packet(c->fd, reply) != 0)
+			dead_client(c);
+		else
+			condlog(4, "cli[%d]: Reply [%d bytes]", c->fd, rlen);
+		FREE(reply);
+		reply = NULL;
+	}
+	FREE(inbuf);
+}
+
 /*
  * entry point
  */
 void *uxsock_listen(long ux_sock, void *trigger_data)
 {
-	int rlen;
-	char *inbuf;
-	char *reply;
 	sigset_t mask;
 	int max_pfds = MIN_POLLS + POLLFDS_BASE;
 	/* conf->sequence_nr will be 1 when uxsock_listen is first called */
@@ -504,8 +532,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 		/* see if a client wants to speak to us */
 		for (i = POLLFDS_BASE; i < n_pfds; i++) {
 			if (polls[i].revents & (POLLIN|POLLHUP|POLLERR)) {
-				struct timespec start_time;
-
 				c = NULL;
 				pthread_mutex_lock(&client_lock);
 				list_for_each_entry(tmp, &clients, node) {
@@ -526,36 +552,7 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 					dead_client(c);
 					continue;
 				}
-				get_monotonic_time(&start_time);
-				if (recv_packet_from_client(c->fd, &inbuf,
-							    uxsock_timeout)
-				    != 0) {
-					dead_client(c);
-					continue;
-				}
-				if (!inbuf) {
-					condlog(4, "recv_packet_from_client "
-						"get null request");
-					continue;
-				}
-				condlog(4, "cli[%d]: Got request [%s]",
-					polls[i].fd, inbuf);
-				uxsock_trigger(inbuf, &reply, &rlen,
-					       _socket_client_is_root(c->fd),
-					       trigger_data);
-				if (reply) {
-					if (send_packet(c->fd,
-							reply) != 0) {
-						dead_client(c);
-					} else {
-						condlog(4, "cli[%d]: "
-							"Reply [%d bytes]",
-							polls[i].fd, rlen);
-					}
-					FREE(reply);
-					reply = NULL;
-				}
-				FREE(inbuf);
+				handle_client(c, trigger_data);
 			}
 		}
 		/* see if we got a non-fatal signal */
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 23/48] multipathd: uxlsnr: use main poll loop for receiving

[mwilck]

From: Martin Wilck <mwilck@suse.com>

As a first step towards our state machine, avoid the call to
read_all() via recv_packet_from_client(). handle_client() is now
invoked twice for the same connection. The first time it reads
the command length, and later on it reads the command itself
piece-wise, as sent by the client. This will be just a single
read in most cases, but not always.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 90 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 15 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 2fb23c8..eff4f7b 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -292,6 +292,8 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 		condlog(1, "Multipath configuration updated.\nReload multipathd for changes to take effect");
 }
 
+static const struct timespec ts_zero = { .tv_sec = 0, };
+
 static int parse_cmd (char *cmd, char **reply, int *len, void *data,
 		      int timeout)
 {
@@ -394,23 +396,78 @@ static int uxsock_trigger(char *str, char **reply, int *len,
 	return r;
 }
 
+static void set_client_state(struct client *c, int state)
+{
+	switch(state)
+	{
+	case CLT_RECV:
+		reset_strbuf(&c->reply);
+		memset(c->cmd, '\0', sizeof(c->cmd));
+		c->expires = ts_zero;
+		/* fallthrough */
+	case CLT_SEND:
+		/* reuse these fields for next data transfer */
+		c->len = c->cmd_len = 0;
+		break;
+	default:
+		break;
+	}
+	c->state = state;
+}
+
 static void handle_client(struct client *c, void *trigger_data)
 {
 	int rlen;
-	char *inbuf, *reply;
+	char *reply;
+	ssize_t n;
 
-	if (recv_packet_from_client(c->fd, &inbuf, uxsock_timeout) != 0) {
-		dead_client(c);
-		return;
+	switch (c->state) {
+	case CLT_RECV:
+		if (c->cmd_len == 0) {
+			/*
+			 * We got POLLIN; assume that at least the length can
+			 * be read immediately.
+			 */
+			get_monotonic_time(&c->expires);
+			c->expires.tv_sec += uxsock_timeout / 1000;
+			c->expires.tv_nsec += (uxsock_timeout % 1000) * 1000000;
+			normalize_timespec(&c->expires);
+			n = mpath_recv_reply_len(c->fd, 0);
+			if (n == -1) {
+				condlog(1, "%s: cli[%d]: failed to receive reply len",
+					__func__, c->fd);
+				c->error = -ECONNRESET;
+			} else if (n > _MAX_CMD_LEN) {
+				condlog(1, "%s: cli[%d]: overlong command (%zd bytes)",
+					__func__, c->fd, n);
+				c->error = -ECONNRESET;
+			} else {
+				c->cmd_len = n;
+				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
+			}
+			/* poll for data */
+			return;
+		} else if (c->len < c->cmd_len) {
+			n = recv(c->fd, c->cmd + c->len, c->cmd_len - c->len, 0);
+			if (n <= 0 && errno != EINTR && errno != EAGAIN) {
+				condlog(1, "%s: cli[%d]: error in recv: %m",
+					__func__, c->fd);
+				c->error = -ECONNRESET;
+				return;
+			}
+			c->len += n;
+			if (c->len < c->cmd_len)
+				/* continue polling */
+				return;
+			set_client_state(c, CLT_PARSE);
+		}
+		break;
+	default:
+		break;
 	}
 
-	if (!inbuf) {
-		condlog(4, "recv_packet_from_client get null request");
-		return;
-	}
-
-	condlog(4, "cli[%d]: Got request [%s]", c->fd, inbuf);
-	uxsock_trigger(inbuf, &reply, &rlen,
+	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
+	uxsock_trigger(c->cmd, &reply, &rlen,
 		       _socket_client_is_root(c->fd),
 		       trigger_data);
 
@@ -418,11 +475,12 @@ static void handle_client(struct client *c, void *trigger_data)
 		if (send_packet(c->fd, reply) != 0)
 			dead_client(c);
 		else
-			condlog(4, "cli[%d]: Reply [%d bytes]", c->fd, rlen);
-		FREE(reply);
-		reply = NULL;
+			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
+				get_strbuf_len(&c->reply) + 1);
+		reset_strbuf(&c->reply);
 	}
-	FREE(inbuf);
+
+	set_client_state(c, CLT_RECV);
 }
 
 /*
@@ -553,6 +611,8 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 					continue;
 				}
 				handle_client(c, trigger_data);
+				if (c->error == -ECONNRESET)
+					dead_client(c);
 			}
 		}
 		/* see if we got a non-fatal signal */
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 24/48] multipathd: use strbuf in cli_handler functions

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This allows us to simplify callers by not having to track the
reply length separately.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/cli.c          |   7 +-
 multipathd/cli.h          |   6 +-
 multipathd/cli_handlers.c | 316 +++++++++++++++-----------------------
 multipathd/uxlsnr.c       |  49 +++---
 4 files changed, 152 insertions(+), 226 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index eb94c75..1c6351e 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -417,13 +417,10 @@ do_genhelp(struct strbuf *reply, const char *cmd, int error) {
 }
 
 
-char *genhelp_handler(const char *cmd, int error)
+void genhelp_handler(const char *cmd, int error, struct strbuf *reply)
 {
-	STRBUF_ON_STACK(reply);
-
-	if (do_genhelp(&reply, cmd, error) == -1)
+	if (do_genhelp(reply, cmd, error) == -1)
 		condlog(0, "genhelp_handler: out of memory");
-	return steal_strbuf_str(&reply);
 }
 
 char *
diff --git a/multipathd/cli.h b/multipathd/cli.h
index f795826..b05746f 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -124,7 +124,9 @@ struct key {
 	int has_param;
 };
 
-typedef int (cli_handler)(void *keywords, char **reply, int *len, void *data);
+struct strbuf;
+
+typedef int (cli_handler)(void *keywords, struct strbuf *reply, void *data);
 
 struct handler {
 	uint64_t fingerprint;
@@ -139,7 +141,7 @@ int __set_handler_callback (uint64_t fp, cli_handler *fn, bool locked);
 
 int get_cmdvec (char *cmd, vector *v);
 struct handler *find_handler_for_cmdvec(const struct _vector *v);
-char *genhelp_handler (const char *cmd, int error);
+void genhelp_handler (const char *cmd, int error, struct strbuf *reply);
 
 int load_keys (void);
 char * get_keyparam (vector v, uint64_t code);
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 1a9c822..f59db3a 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -33,17 +33,9 @@
 #include "strbuf.h"
 #include "cli_handlers.h"
 
-#define SET_REPLY_AND_LEN(__rep, __len, string_literal)			\
-	do {								\
-		*(__rep) = strdup(string_literal);			\
-		*(__len) = *(__rep) ? sizeof(string_literal) : 0;	\
-	} while (0)
-
 static int
-show_paths (char ** r, int * len, struct vectors * vecs, char * style,
-	    int pretty)
+show_paths (struct strbuf *reply, struct vectors *vecs, char *style, int pretty)
 {
-	STRBUF_ON_STACK(reply);
 	int i;
 	struct path * pp;
 	int hdr_len = 0;
@@ -51,61 +43,49 @@ show_paths (char ** r, int * len, struct vectors * vecs, char * style,
 	get_path_layout(vecs->pathvec, 1);
 	foreign_path_layout();
 
-	if (pretty && (hdr_len = snprint_path_header(&reply, style)) < 0)
+	if (pretty && (hdr_len = snprint_path_header(reply, style)) < 0)
 		return 1;
 
 	vector_foreach_slot(vecs->pathvec, pp, i) {
-		if (snprint_path(&reply, style, pp, pretty) < 0)
+		if (snprint_path(reply, style, pp, pretty) < 0)
 			return 1;
 	}
-	if (snprint_foreign_paths(&reply, style, pretty) < 0)
+	if (snprint_foreign_paths(reply, style, pretty) < 0)
 		return 1;
 
-	if (pretty && get_strbuf_len(&reply) == (size_t)hdr_len)
+	if (pretty && get_strbuf_len(reply) == (size_t)hdr_len)
 		/* No output - clear header */
-		truncate_strbuf(&reply, 0);
+		truncate_strbuf(reply, 0);
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_path (char ** r, int * len, struct vectors * vecs, struct path *pp,
-	   char * style)
+show_path (struct strbuf *reply, struct vectors *vecs, struct path *pp,
+	   char *style)
 {
-	STRBUF_ON_STACK(reply);
-
 	get_path_layout(vecs->pathvec, 1);
-	if (snprint_path(&reply, style, pp, 0) < 0)
+	if (snprint_path(reply, style, pp, 0) < 0)
 		return 1;
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
-
 	return 0;
 }
 
 static int
-show_map_topology (char ** r, int * len, struct multipath * mpp,
-		   struct vectors * vecs)
+show_map_topology (struct strbuf *reply, struct multipath *mpp,
+		   struct vectors *vecs)
 {
-	STRBUF_ON_STACK(reply);
-
 	if (update_multipath(vecs, mpp->alias, 0))
 		return 1;
 
-	if (snprint_multipath_topology(&reply, mpp, 2) < 0)
+	if (snprint_multipath_topology(reply, mpp, 2) < 0)
 		return 1;
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 
 	return 0;
 }
 
 static int
-show_maps_topology (char ** r, int * len, struct vectors * vecs)
+show_maps_topology (struct strbuf *reply, struct vectors * vecs)
 {
-	STRBUF_ON_STACK(reply);
 	int i;
 	struct multipath * mpp;
 
@@ -117,21 +97,18 @@ show_maps_topology (char ** r, int * len, struct vectors * vecs)
 			i--;
 			continue;
 		}
-		if (snprint_multipath_topology(&reply, mpp, 2) < 0)
+		if (snprint_multipath_topology(reply, mpp, 2) < 0)
 			return 1;
 	}
-	if (snprint_foreign_topology(&reply, 2) < 0)
+	if (snprint_foreign_topology(reply, 2) < 0)
 		return 1;
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_maps_json (char ** r, int * len, struct vectors * vecs)
+show_maps_json (struct strbuf *reply, struct vectors * vecs)
 {
-	STRBUF_ON_STACK(reply);
 	int i;
 	struct multipath * mpp;
 
@@ -141,45 +118,38 @@ show_maps_json (char ** r, int * len, struct vectors * vecs)
 		}
 	}
 
-	if (snprint_multipath_topology_json(&reply, vecs) < 0)
+	if (snprint_multipath_topology_json(reply, vecs) < 0)
 		return 1;
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_map_json (char ** r, int * len, struct multipath * mpp,
-		   struct vectors * vecs)
+show_map_json (struct strbuf *reply, struct multipath * mpp,
+	       struct vectors * vecs)
 {
-	STRBUF_ON_STACK(reply);
-
 	if (update_multipath(vecs, mpp->alias, 0))
 		return 1;
 
-	if (snprint_multipath_map_json(&reply, mpp) < 0)
+	if (snprint_multipath_map_json(reply, mpp) < 0)
 		return 1;
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_config (char ** r, int * len, const struct _vector *hwtable,
+show_config (struct strbuf *reply, const struct _vector *hwtable,
 	     const struct _vector *mpvec)
 {
 	struct config *conf;
-	char *reply;
+	int rc;
 
 	conf = get_multipath_config();
 	pthread_cleanup_push(put_multipath_config, conf);
-	reply = snprint_config(conf, len, hwtable, mpvec);
+	rc = __snprint_config(conf, reply, hwtable, mpvec);
 	pthread_cleanup_pop(1);
-	if (reply == NULL)
+	if (rc < 0)
 		return 1;
-	*r = reply;
 	return 0;
 }
 
@@ -195,11 +165,11 @@ reset_stats(struct multipath * mpp)
 }
 
 static int
-cli_list_config (void * v, char ** reply, int * len, void * data)
+cli_list_config (void *v, struct strbuf *reply, void *data)
 {
 	condlog(3, "list config (operator)");
 
-	return show_config(reply, len, NULL, NULL);
+	return show_config(reply, NULL, NULL);
 }
 
 static void v_free(void *x)
@@ -208,9 +178,9 @@ static void v_free(void *x)
 }
 
 static int
-cli_list_config_local (void * v, char ** reply, int * len, void * data)
+cli_list_config_local (void *v, struct strbuf *reply, void *data)
 {
-	struct vectors * vecs = (struct vectors *)data;
+	struct vectors *vecs = (struct vectors *)data;
 	vector hwes;
 	int ret;
 
@@ -218,45 +188,45 @@ cli_list_config_local (void * v, char ** reply, int * len, void * data)
 
 	hwes = get_used_hwes(vecs->pathvec);
 	pthread_cleanup_push(v_free, hwes);
-	ret = show_config(reply, len, hwes, vecs->mpvec);
+	ret = show_config(reply, hwes, vecs->mpvec);
 	pthread_cleanup_pop(1);
 	return ret;
 }
 
 static int
-cli_list_paths (void * v, char ** reply, int * len, void * data)
+cli_list_paths (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list paths (operator)");
 
-	return show_paths(reply, len, vecs, PRINT_PATH_CHECKER, 1);
+	return show_paths(reply, vecs, PRINT_PATH_CHECKER, 1);
 }
 
 static int
-cli_list_paths_fmt (void * v, char ** reply, int * len, void * data)
+cli_list_paths_fmt (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * fmt = get_keyparam(v, FMT);
 
 	condlog(3, "list paths (operator)");
 
-	return show_paths(reply, len, vecs, fmt, 1);
+	return show_paths(reply, vecs, fmt, 1);
 }
 
 static int
-cli_list_paths_raw (void * v, char ** reply, int * len, void * data)
+cli_list_paths_raw (void *v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * fmt = get_keyparam(v, FMT);
 
 	condlog(3, "list paths (operator)");
 
-	return show_paths(reply, len, vecs, fmt, 0);
+	return show_paths(reply, vecs, fmt, 0);
 }
 
 static int
-cli_list_path (void * v, char ** reply, int * len, void * data)
+cli_list_path (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -269,11 +239,11 @@ cli_list_path (void * v, char ** reply, int * len, void * data)
 	if (!pp)
 		return 1;
 
-	return show_path(reply, len, vecs, pp, "%o");
+	return show_path(reply, vecs, pp, "%o");
 }
 
 static int
-cli_list_map_topology (void * v, char ** reply, int * len, void * data)
+cli_list_map_topology (void *v, struct strbuf *reply, void *data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -288,21 +258,21 @@ cli_list_map_topology (void * v, char ** reply, int * len, void * data)
 
 	condlog(3, "list multipath %s (operator)", param);
 
-	return show_map_topology(reply, len, mpp, vecs);
+	return show_map_topology(reply, mpp, vecs);
 }
 
 static int
-cli_list_maps_topology (void * v, char ** reply, int * len, void * data)
+cli_list_maps_topology (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list multipaths (operator)");
 
-	return show_maps_topology(reply, len, vecs);
+	return show_maps_topology(reply, vecs);
 }
 
 static int
-cli_list_map_json (void * v, char ** reply, int * len, void * data)
+cli_list_map_json (void *v, struct strbuf *reply, void *data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -317,78 +287,61 @@ cli_list_map_json (void * v, char ** reply, int * len, void * data)
 
 	condlog(3, "list multipath json %s (operator)", param);
 
-	return show_map_json(reply, len, mpp, vecs);
+	return show_map_json(reply, mpp, vecs);
 }
 
 static int
-cli_list_maps_json (void * v, char ** reply, int * len, void * data)
+cli_list_maps_json (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list multipaths json (operator)");
 
-	return show_maps_json(reply, len, vecs);
+	return show_maps_json(reply, vecs);
 }
 
 static int
-cli_list_wildcards (void * v, char ** reply, int * len, void * data)
+cli_list_wildcards (void *v, struct strbuf *reply, void *data)
 {
-	STRBUF_ON_STACK(buf);
-
-	if (snprint_wildcards(&buf) < 0)
+	if (snprint_wildcards(reply) < 0)
 		return 1;
 
-	*len = get_strbuf_len(&buf) + 1;
-	*reply = steal_strbuf_str(&buf);
 	return 0;
 }
 
 static int
-show_status (char ** r, int *len, struct vectors * vecs)
+show_status (struct strbuf *reply, struct vectors *vecs)
 {
-	STRBUF_ON_STACK(reply);
-
-	if (snprint_status(&reply, vecs) < 0)
+	if (snprint_status(reply, vecs) < 0)
 		return 1;
 
-	*len = get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_daemon (char ** r, int *len)
+show_daemon (struct strbuf *reply)
 {
-	STRBUF_ON_STACK(reply);
-
-	if (print_strbuf(&reply, "pid %d %s\n",
+	if (print_strbuf(reply, "pid %d %s\n",
 			 daemon_pid, daemon_status()) < 0)
 		return 1;
 
-	*len = get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_map (char ** r, int *len, struct multipath * mpp, char * style,
+show_map (struct strbuf *reply, struct multipath *mpp, char *style,
 	  int pretty)
 {
-	STRBUF_ON_STACK(reply);
-
-	if (snprint_multipath(&reply, style, mpp, pretty) < 0)
+	if (snprint_multipath(reply, style, mpp, pretty) < 0)
 		return 1;
 
-	*len = get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-show_maps (char ** r, int *len, struct vectors * vecs, char * style,
+show_maps (struct strbuf *reply, struct vectors *vecs, char *style,
 	   int pretty)
 {
-	STRBUF_ON_STACK(reply);
 	int i;
 	struct multipath * mpp;
 	int hdr_len = 0;
@@ -396,7 +349,7 @@ show_maps (char ** r, int *len, struct vectors * vecs, char * style,
 	get_multipath_layout(vecs->mpvec, 1);
 	foreign_multipath_layout();
 
-	if (pretty && (hdr_len = snprint_multipath_header(&reply, style)) < 0)
+	if (pretty && (hdr_len = snprint_multipath_header(reply, style)) < 0)
 		return 1;
 
 	vector_foreach_slot(vecs->mpvec, mpp, i) {
@@ -404,45 +357,43 @@ show_maps (char ** r, int *len, struct vectors * vecs, char * style,
 			i--;
 			continue;
 		}
-		if (snprint_multipath(&reply, style, mpp, pretty) < 0)
+		if (snprint_multipath(reply, style, mpp, pretty) < 0)
 			return 1;
 	}
-	if (snprint_foreign_multipaths(&reply, style, pretty) < 0)
+	if (snprint_foreign_multipaths(reply, style, pretty) < 0)
 		return 1;
 
-	if (pretty && get_strbuf_len(&reply) == (size_t)hdr_len)
+	if (pretty && get_strbuf_len(reply) == (size_t)hdr_len)
 		/* No output - clear header */
-		truncate_strbuf(&reply, 0);
+		truncate_strbuf(reply, 0);
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-cli_list_maps_fmt (void * v, char ** reply, int * len, void * data)
+cli_list_maps_fmt (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * fmt = get_keyparam(v, FMT);
 
 	condlog(3, "list maps (operator)");
 
-	return show_maps(reply, len, vecs, fmt, 1);
+	return show_maps(reply, vecs, fmt, 1);
 }
 
 static int
-cli_list_maps_raw (void * v, char ** reply, int * len, void * data)
+cli_list_maps_raw (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * fmt = get_keyparam(v, FMT);
 
 	condlog(3, "list maps (operator)");
 
-	return show_maps(reply, len, vecs, fmt, 0);
+	return show_maps(reply, vecs, fmt, 0);
 }
 
 static int
-cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
+cli_list_map_fmt (void *v, struct strbuf *reply, void *data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -458,59 +409,59 @@ cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
 
 	condlog(3, "list map %s fmt %s (operator)", param, fmt);
 
-	return show_map(reply, len, mpp, fmt, 1);
+	return show_map(reply, mpp, fmt, 1);
 }
 
 static int
-cli_list_maps (void * v, char ** reply, int * len, void * data)
+cli_list_maps (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list maps (operator)");
 
-	return show_maps(reply, len, vecs, PRINT_MAP_NAMES, 1);
+	return show_maps(reply, vecs, PRINT_MAP_NAMES, 1);
 }
 
 static int
-cli_list_status (void * v, char ** reply, int * len, void * data)
+cli_list_status (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list status (operator)");
 
-	return show_status(reply, len, vecs);
+	return show_status(reply, vecs);
 }
 
 static int
-cli_list_maps_status (void * v, char ** reply, int * len, void * data)
+cli_list_maps_status (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list maps status (operator)");
 
-	return show_maps(reply, len, vecs, PRINT_MAP_STATUS, 1);
+	return show_maps(reply, vecs, PRINT_MAP_STATUS, 1);
 }
 
 static int
-cli_list_maps_stats (void * v, char ** reply, int * len, void * data)
+cli_list_maps_stats (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list maps stats (operator)");
 
-	return show_maps(reply, len, vecs, PRINT_MAP_STATS, 1);
+	return show_maps(reply, vecs, PRINT_MAP_STATS, 1);
 }
 
 static int
-cli_list_daemon (void * v, char ** reply, int * len, void * data)
+cli_list_daemon (void *v, struct strbuf *reply, void *data)
 {
 	condlog(3, "list daemon (operator)");
 
-	return show_daemon(reply, len);
+	return show_daemon(reply);
 }
 
 static int
-cli_reset_maps_stats (void * v, char ** reply, int * len, void * data)
+cli_reset_maps_stats (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	int i;
@@ -525,7 +476,7 @@ cli_reset_maps_stats (void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_reset_map_stats (void * v, char ** reply, int * len, void * data)
+cli_reset_map_stats (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	struct multipath * mpp;
@@ -543,7 +494,7 @@ cli_reset_map_stats (void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_add_path (void * v, char ** reply, int * len, void * data)
+cli_add_path (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -653,13 +604,13 @@ cli_add_path (void * v, char ** reply, int * len, void * data)
 	}
 	return ev_add_path(pp, vecs, 1);
 blacklisted:
-	SET_REPLY_AND_LEN(reply, len, "blacklisted\n");
+	append_strbuf_str(reply, "blacklisted\n");
 	condlog(2, "%s: path blacklisted", param);
 	return 0;
 }
 
 static int
-cli_del_path (void * v, char ** reply, int * len, void * data)
+cli_del_path (void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -675,14 +626,14 @@ cli_del_path (void * v, char ** reply, int * len, void * data)
 	}
 	ret = ev_remove_path(pp, vecs, 1);
 	if (ret == REMOVE_PATH_DELAY)
-		SET_REPLY_AND_LEN(reply, len, "delayed\n");
+		append_strbuf_str(reply, "delayed\n");
 	else if (ret == REMOVE_PATH_MAP_ERROR)
-		SET_REPLY_AND_LEN(reply, len, "map reload error. removed\n");
+		append_strbuf_str(reply, "map reload error. removed\n");
 	return (ret == REMOVE_PATH_FAILURE);
 }
 
 static int
-cli_add_map (void * v, char ** reply, int * len, void * data)
+cli_add_map (void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, MAP);
@@ -702,7 +653,7 @@ cli_add_map (void * v, char ** reply, int * len, void * data)
 		invalid = 1;
 	pthread_cleanup_pop(1);
 	if (invalid) {
-		SET_REPLY_AND_LEN(reply, len, "blacklisted\n");
+		append_strbuf_str(reply, "blacklisted\n");
 		condlog(2, "%s: map blacklisted", param);
 		return 1;
 	}
@@ -742,7 +693,7 @@ cli_add_map (void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_del_map (void * v, char ** reply, int * len, void * data)
+cli_del_map (void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, MAP);
@@ -764,14 +715,14 @@ cli_del_map (void * v, char ** reply, int * len, void * data)
 	}
 	rc = ev_remove_map(param, alias, minor, vecs);
 	if (rc == 2)
-		*reply = strdup("delayed");
+		append_strbuf_str(reply, "delayed");
 
 	FREE(alias);
 	return rc;
 }
 
 static int
-cli_del_maps (void *v, char **reply, int *len, void *data)
+cli_del_maps (void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	struct multipath *mpp;
@@ -790,7 +741,7 @@ cli_del_maps (void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_reload(void *v, char **reply, int *len, void *data)
+cli_reload(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
@@ -843,7 +794,7 @@ static int resize_map(struct multipath *mpp, unsigned long long size,
 }
 
 static int
-cli_resize(void *v, char **reply, int *len, void *data)
+cli_resize(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
@@ -908,7 +859,7 @@ cli_resize(void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_force_no_daemon_q(void * v, char ** reply, int * len, void * data)
+cli_force_no_daemon_q(void * v, struct strbuf *reply, void * data)
 {
 	struct config *conf;
 
@@ -921,7 +872,7 @@ cli_force_no_daemon_q(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_restore_no_daemon_q(void * v, char ** reply, int * len, void * data)
+cli_restore_no_daemon_q(void * v, struct strbuf *reply, void * data)
 {
 	struct config *conf;
 
@@ -934,7 +885,7 @@ cli_restore_no_daemon_q(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_restore_queueing(void *v, char **reply, int *len, void *data)
+cli_restore_queueing(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
@@ -975,7 +926,7 @@ cli_restore_queueing(void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_restore_all_queueing(void *v, char **reply, int *len, void *data)
+cli_restore_all_queueing(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	struct multipath *mpp;
@@ -997,7 +948,7 @@ cli_restore_all_queueing(void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_disable_queueing(void *v, char **reply, int *len, void *data)
+cli_disable_queueing(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
@@ -1026,7 +977,7 @@ cli_disable_queueing(void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_disable_all_queueing(void *v, char **reply, int *len, void *data)
+cli_disable_all_queueing(void *v, struct strbuf *reply, void *data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	struct multipath *mpp;
@@ -1045,7 +996,7 @@ cli_disable_all_queueing(void *v, char **reply, int *len, void *data)
 }
 
 static int
-cli_switch_group(void * v, char ** reply, int * len, void * data)
+cli_switch_group(void * v, struct strbuf *reply, void * data)
 {
 	char * mapname = get_keyparam(v, MAP);
 	int groupnum = atoi(get_keyparam(v, GROUP));
@@ -1057,7 +1008,7 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_reconfigure(void * v, char ** reply, int * len, void * data)
+cli_reconfigure(void * v, struct strbuf *reply, void * data)
 {
 	condlog(2, "reconfigure (operator)");
 
@@ -1066,7 +1017,7 @@ cli_reconfigure(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_suspend(void * v, char ** reply, int * len, void * data)
+cli_suspend(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, MAP);
@@ -1096,7 +1047,7 @@ cli_suspend(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_resume(void * v, char ** reply, int * len, void * data)
+cli_resume(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, MAP);
@@ -1128,7 +1079,7 @@ cli_resume(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_reinstate(void * v, char ** reply, int * len, void * data)
+cli_reinstate(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1151,7 +1102,7 @@ cli_reinstate(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_reassign (void * v, char ** reply, int * len, void * data)
+cli_reassign (void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, MAP);
@@ -1175,7 +1126,7 @@ cli_reassign (void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_fail(void * v, char ** reply, int * len, void * data)
+cli_fail(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1204,72 +1155,65 @@ cli_fail(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-show_blacklist (char ** r, int * len)
+show_blacklist (struct strbuf *reply)
 {
-	STRBUF_ON_STACK(reply);
 	struct config *conf;
 	bool fail;
 
 	conf = get_multipath_config();
 	pthread_cleanup_push(put_multipath_config, conf);
-	fail = snprint_blacklist_report(conf, &reply) < 0;
+	fail = snprint_blacklist_report(conf, reply) < 0;
 	pthread_cleanup_pop(1);
 
 	if (fail)
 		return 1;
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
 	return 0;
 }
 
 static int
-cli_list_blacklist (void * v, char ** reply, int * len, void * data)
+cli_list_blacklist (void * v, struct strbuf *reply, void * data)
 {
 	condlog(3, "list blacklist (operator)");
 
-	return show_blacklist(reply, len);
+	return show_blacklist(reply);
 }
 
 static int
-show_devices (char ** r, int * len, struct vectors *vecs)
+show_devices (struct strbuf *reply, struct vectors *vecs)
 {
-	STRBUF_ON_STACK(reply);
 	struct config *conf;
 	bool fail;
 
 	conf = get_multipath_config();
 	pthread_cleanup_push(put_multipath_config, conf);
-	fail = snprint_devices(conf, &reply, vecs) < 0;
+	fail = snprint_devices(conf, reply, vecs) < 0;
 	pthread_cleanup_pop(1);
 
 	if (fail)
 		return 1;
 
-	*len = (int)get_strbuf_len(&reply) + 1;
-	*r = steal_strbuf_str(&reply);
-
 	return 0;
 }
 
 static int
-cli_list_devices (void * v, char ** reply, int * len, void * data)
+cli_list_devices (void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 
 	condlog(3, "list devices (operator)");
 
-	return show_devices(reply, len, vecs);
+	return show_devices(reply, vecs);
 }
 
 static int
-cli_quit (void * v, char ** reply, int * len, void * data)
+cli_quit (void * v, struct strbuf *reply, void * data)
 {
 	return 0;
 }
 
 static int
-cli_shutdown (void * v, char ** reply, int * len, void * data)
+cli_shutdown (void * v, struct strbuf *reply, void * data)
 {
 	condlog(3, "shutdown (operator)");
 	exit_daemon();
@@ -1277,7 +1221,7 @@ cli_shutdown (void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_getprstatus (void * v, char ** reply, int * len, void * data)
+cli_getprstatus (void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1292,17 +1236,16 @@ cli_getprstatus (void * v, char ** reply, int * len, void * data)
 
 	condlog(3, "%s: prflag = %u", param, (unsigned int)mpp->prflag);
 
-	*len = asprintf(reply, "%d", mpp->prflag);
-	if (*len < 0)
+	if (print_strbuf(reply, "%d", mpp->prflag) < 0)
 		return 1;
 
-	condlog(3, "%s: reply = %s", param, *reply);
+	condlog(3, "%s: reply = %s", param, get_strbuf_str(reply));
 
 	return 0;
 }
 
 static int
-cli_setprstatus(void * v, char ** reply, int * len, void * data)
+cli_setprstatus(void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1325,7 +1268,7 @@ cli_setprstatus(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_unsetprstatus(void * v, char ** reply, int * len, void * data)
+cli_unsetprstatus(void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1347,7 +1290,7 @@ cli_unsetprstatus(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_getprkey(void * v, char ** reply, int * len, void * data)
+cli_getprkey(void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1361,25 +1304,20 @@ cli_getprkey(void * v, char ** reply, int * len, void * data)
 	if (!mpp)
 		return 1;
 
-	*reply = malloc(26);
-	if (!*reply)
-		return 1;
-
 	key = get_be64(mpp->reservation_key);
 	if (!key) {
-		sprintf(*reply, "none\n");
-		*len = sizeof("none\n");
+		append_strbuf_str(reply, "none\n");
 		return 0;
 	}
 
-	/* This snprintf() can't overflow - PRIx64 needs max 16 chars */
-	*len = snprintf(*reply, 26, "0x%" PRIx64 "%s\n", key,
-			mpp->sa_flags & MPATH_F_APTPL_MASK ? ":aptpl" : "") + 1;
+	if (print_strbuf(reply, "0x%" PRIx64 "%s\n", key,
+			 mpp->sa_flags & MPATH_F_APTPL_MASK ? ":aptpl" : "") < 0)
+		return 1;
 	return 0;
 }
 
 static int
-cli_unsetprkey(void * v, char ** reply, int * len, void * data)
+cli_unsetprkey(void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1403,7 +1341,7 @@ cli_unsetprkey(void * v, char ** reply, int * len, void * data)
 }
 
 static int
-cli_setprkey(void * v, char ** reply, int * len, void * data)
+cli_setprkey(void * v, struct strbuf *reply, void * data)
 {
 	struct multipath * mpp;
 	struct vectors * vecs = (struct vectors *)data;
@@ -1434,7 +1372,7 @@ cli_setprkey(void * v, char ** reply, int * len, void * data)
 	return ret;
 }
 
-static int cli_set_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_set_marginal(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1461,7 +1399,7 @@ static int cli_set_marginal(void * v, char ** reply, int * len, void * data)
 	return reload_and_sync_map(pp->mpp, vecs, 0);
 }
 
-static int cli_unset_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_unset_marginal(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * param = get_keyparam(v, PATH);
@@ -1488,7 +1426,7 @@ static int cli_unset_marginal(void * v, char ** reply, int * len, void * data)
 	return reload_and_sync_map(pp->mpp, vecs, 0);
 }
 
-static int cli_unset_all_marginal(void * v, char ** reply, int * len, void * data)
+static int cli_unset_all_marginal(void * v, struct strbuf *reply, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
 	char * mapname = get_keyparam(v, MAP);
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index eff4f7b..9a945ea 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -294,7 +294,7 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 
 static const struct timespec ts_zero = { .tv_sec = 0, };
 
-static int parse_cmd (char *cmd, char **reply, int *len, void *data,
+static int parse_cmd (char *cmd, struct strbuf *reply, void *data,
 		      int timeout)
 {
 	int r;
@@ -305,10 +305,9 @@ static int parse_cmd (char *cmd, char **reply, int *len, void *data,
 	r = get_cmdvec(cmd, &cmdvec);
 
 	if (r) {
-		*reply = genhelp_handler(cmd, r);
-		if (*reply == NULL)
+		genhelp_handler(cmd, r, reply);
+		if (get_strbuf_len(reply) == 0)
 			return EINVAL;
-		*len = strlen(*reply) + 1;
 		return 0;
 	}
 
@@ -316,10 +315,9 @@ static int parse_cmd (char *cmd, char **reply, int *len, void *data,
 
 	if (!h || !h->fn) {
 		free_keys(cmdvec);
-		*reply = genhelp_handler(cmd, EINVAL);
-		if (*reply == NULL)
+		genhelp_handler(cmd, EINVAL, reply);
+		if (get_strbuf_len(reply) == 0)
 			return EINVAL;
-		*len = strlen(*reply) + 1;
 		return 0;
 	}
 
@@ -345,50 +343,42 @@ static int parse_cmd (char *cmd, char **reply, int *len, void *data,
 		if (r == 0) {
 			locked = 1;
 			pthread_testcancel();
-			r = h->fn(cmdvec, reply, len, data);
+			r = h->fn(cmdvec, reply, data);
 		}
 		pthread_cleanup_pop(locked);
 	} else
-		r = h->fn(cmdvec, reply, len, data);
+		r = h->fn(cmdvec, reply, data);
 	free_keys(cmdvec);
 
 	return r;
 }
 
-static int uxsock_trigger(char *str, char **reply, int *len,
+static int uxsock_trigger(char *str, struct strbuf *reply,
 			  bool is_root, void *trigger_data)
 {
 	struct vectors * vecs;
 	int r;
 
-	*reply = NULL;
-	*len = 0;
 	vecs = (struct vectors *)trigger_data;
 
 	if ((str != NULL) && (is_root == false) &&
 	    (strncmp(str, "list", strlen("list")) != 0) &&
 	    (strncmp(str, "show", strlen("show")) != 0)) {
-		*reply = STRDUP("permission deny: need to be root");
-		if (*reply)
-			*len = strlen(*reply) + 1;
+		append_strbuf_str(reply, "permission deny: need to be root");
 		return 1;
 	}
 
-	r = parse_cmd(str, reply, len, vecs, uxsock_timeout / 1000);
+	r = parse_cmd(str, reply, vecs, uxsock_timeout / 1000);
 
 	if (r > 0) {
 		if (r == ETIMEDOUT)
-			*reply = STRDUP("timeout\n");
+			append_strbuf_str(reply, "timeout\n");
 		else
-			*reply = STRDUP("fail\n");
-		if (*reply)
-			*len = strlen(*reply) + 1;
+			append_strbuf_str(reply, "fail\n");
 		r = 1;
 	}
-	else if (!r && *len == 0) {
-		*reply = STRDUP("ok\n");
-		if (*reply)
-			*len = strlen(*reply) + 1;
+	else if (!r && get_strbuf_len(reply) == 0) {
+		append_strbuf_str(reply, "ok\n");
 		r = 0;
 	}
 	/* else if (r < 0) leave *reply alone */
@@ -417,8 +407,6 @@ static void set_client_state(struct client *c, int state)
 
 static void handle_client(struct client *c, void *trigger_data)
 {
-	int rlen;
-	char *reply;
 	ssize_t n;
 
 	switch (c->state) {
@@ -467,12 +455,13 @@ static void handle_client(struct client *c, void *trigger_data)
 	}
 
 	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
-	uxsock_trigger(c->cmd, &reply, &rlen,
-		       _socket_client_is_root(c->fd),
+	uxsock_trigger(c->cmd, &c->reply, _socket_client_is_root(c->fd),
 		       trigger_data);
 
-	if (reply) {
-		if (send_packet(c->fd, reply) != 0)
+	if (get_strbuf_len(&c->reply) > 0) {
+		const char *buf = get_strbuf_str(&c->reply);
+
+		if (send_packet(c->fd, buf) != 0)
 			dead_client(c);
 		else
 			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 25/48] multipathd: uxlsnr: check root on connection startup

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The SO_PEERCRED socket option returns "the credentials that were
in effect at the time of the call to connect(2)" (see unix(7)).
So we might as well fetch these credentials at that time.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 9a945ea..9cf6964 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -92,8 +92,6 @@ static struct pollfd *polls;
 static int notify_fd = -1;
 static char *watch_config_dir;
 
-static bool _socket_client_is_root(int fd);
-
 static bool _socket_client_is_root(int fd)
 {
 	socklen_t len = 0;
@@ -132,6 +130,7 @@ static void new_client(int ux_sock)
 	INIT_LIST_HEAD(&c->node);
 	c->fd = fd;
 	c->state = CLT_RECV;
+	c->is_root = _socket_client_is_root(c->fd);
 
 	/* put it in our linked list */
 	pthread_mutex_lock(&client_lock);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 26/48] multipathd: uxlsnr: pass struct client to uxsock_trigger() and parse_cmd()

[mwilck]

From: Martin Wilck <mwilck@suse.com>

As a next step towards the state machine, give the handler functions
access to the state of the client connection.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 61 +++++++++++++++++++++------------------------
 1 file changed, 29 insertions(+), 32 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 9cf6964..65cb5ca 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -293,31 +293,28 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 
 static const struct timespec ts_zero = { .tv_sec = 0, };
 
-static int parse_cmd (char *cmd, struct strbuf *reply, void *data,
-		      int timeout)
+static int parse_cmd (struct client *c, void *data, int timeout)
 {
 	int r;
 	struct handler * h;
-	vector cmdvec = NULL;
 	struct timespec tmo;
 
-	r = get_cmdvec(cmd, &cmdvec);
+	r = get_cmdvec(c->cmd, &c->cmdvec);
 
 	if (r) {
-		genhelp_handler(cmd, r, reply);
-		if (get_strbuf_len(reply) == 0)
+		genhelp_handler(c->cmd, r, &c->reply);
+		if (get_strbuf_len(&c->reply) == 0)
 			return EINVAL;
 		return 0;
 	}
 
-	h = find_handler_for_cmdvec(cmdvec);
+	h = find_handler_for_cmdvec(c->cmdvec);
 
 	if (!h || !h->fn) {
-		free_keys(cmdvec);
-		genhelp_handler(cmd, EINVAL, reply);
-		if (get_strbuf_len(reply) == 0)
-			return EINVAL;
-		return 0;
+		genhelp_handler(c->cmd, EINVAL, &c->reply);
+		if (get_strbuf_len(&c->reply) == 0)
+			r = EINVAL;
+		goto free_cmdvec;
 	}
 
 	/*
@@ -342,46 +339,47 @@ static int parse_cmd (char *cmd, struct strbuf *reply, void *data,
 		if (r == 0) {
 			locked = 1;
 			pthread_testcancel();
-			r = h->fn(cmdvec, reply, data);
+			r = h->fn(c->cmdvec, &c->reply, data);
 		}
 		pthread_cleanup_pop(locked);
 	} else
-		r = h->fn(cmdvec, reply, data);
-	free_keys(cmdvec);
+		r = h->fn(c->cmdvec, &c->reply, data);
+
+free_cmdvec:
+	free_keys(c->cmdvec);
+	c->cmdvec = NULL;
 
 	return r;
 }
 
-static int uxsock_trigger(char *str, struct strbuf *reply,
-			  bool is_root, void *trigger_data)
+static int uxsock_trigger(struct client *c, void *trigger_data)
 {
 	struct vectors * vecs;
-	int r;
+	int r = 1;
 
 	vecs = (struct vectors *)trigger_data;
 
-	if ((str != NULL) && (is_root == false) &&
-	    (strncmp(str, "list", strlen("list")) != 0) &&
-	    (strncmp(str, "show", strlen("show")) != 0)) {
-		append_strbuf_str(reply, "permission deny: need to be root");
-		return 1;
+
+	if (!c->is_root &&
+	    (strncmp(c->cmd, "list", strlen("list")) != 0) &&
+	    (strncmp(c->cmd, "show", strlen("show")) != 0)) {
+		append_strbuf_str(&c->reply, "permission deny: need to be root");
+		return r;
 	}
 
-	r = parse_cmd(str, reply, vecs, uxsock_timeout / 1000);
+	r = parse_cmd(c, vecs, uxsock_timeout / 1000);
 
 	if (r > 0) {
 		if (r == ETIMEDOUT)
-			append_strbuf_str(reply, "timeout\n");
+			append_strbuf_str(&c->reply, "timeout\n");
 		else
-			append_strbuf_str(reply, "fail\n");
-		r = 1;
+			append_strbuf_str(&c->reply, "fail\n");
 	}
-	else if (!r && get_strbuf_len(reply) == 0) {
-		append_strbuf_str(reply, "ok\n");
+	else if (!r && get_strbuf_len(&c->reply) == 0) {
+		append_strbuf_str(&c->reply, "ok\n");
 		r = 0;
 	}
 	/* else if (r < 0) leave *reply alone */
-
 	return r;
 }
 
@@ -454,8 +452,7 @@ static void handle_client(struct client *c, void *trigger_data)
 	}
 
 	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
-	uxsock_trigger(c->cmd, &c->reply, _socket_client_is_root(c->fd),
-		       trigger_data);
+	uxsock_trigger(c, trigger_data);
 
 	if (get_strbuf_len(&c->reply) > 0) {
 		const char *buf = get_strbuf_str(&c->reply);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 27/48] multipathd: uxlsnr: move handler execution to separate function

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Move the actual execution of the handler out of parse_cmd(). For now,
we do it in uxsock_trigger().

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 47 ++++++++++++++++++++++++++++-----------------
 1 file changed, 29 insertions(+), 18 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 65cb5ca..cfff0ae 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -293,11 +293,9 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 
 static const struct timespec ts_zero = { .tv_sec = 0, };
 
-static int parse_cmd (struct client *c, void *data, int timeout)
+static int parse_cmd(struct client *c)
 {
 	int r;
-	struct handler * h;
-	struct timespec tmo;
 
 	r = get_cmdvec(c->cmd, &c->cmdvec);
 
@@ -308,26 +306,35 @@ static int parse_cmd (struct client *c, void *data, int timeout)
 		return 0;
 	}
 
-	h = find_handler_for_cmdvec(c->cmdvec);
+	c->handler = find_handler_for_cmdvec(c->cmdvec);
 
-	if (!h || !h->fn) {
+	if (!c->handler || !c->handler->fn) {
 		genhelp_handler(c->cmd, EINVAL, &c->reply);
 		if (get_strbuf_len(&c->reply) == 0)
 			r = EINVAL;
-		goto free_cmdvec;
+		else
+			r = 0;
 	}
 
-	/*
-	 * execute handler
-	 */
+	return r;
+}
+
+static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
+{
+	int r;
+	struct timespec tmo;
+
+	if (!c->handler)
+		return EINVAL;
+
 	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
 		tmo.tv_sec += timeout;
 	} else {
 		tmo.tv_sec = 0;
 	}
-	if (h->locked) {
+
+	if (c->handler->locked) {
 		int locked = 0;
-		struct vectors * vecs = (struct vectors *)data;
 
 		pthread_cleanup_push(cleanup_lock, &vecs->lock);
 		if (tmo.tv_sec) {
@@ -339,15 +346,11 @@ static int parse_cmd (struct client *c, void *data, int timeout)
 		if (r == 0) {
 			locked = 1;
 			pthread_testcancel();
-			r = h->fn(c->cmdvec, &c->reply, data);
+			r = c->handler->fn(c->cmdvec, &c->reply, vecs);
 		}
 		pthread_cleanup_pop(locked);
 	} else
-		r = h->fn(c->cmdvec, &c->reply, data);
-
-free_cmdvec:
-	free_keys(c->cmdvec);
-	c->cmdvec = NULL;
+		r = c->handler->fn(c->cmdvec, &c->reply, vecs);
 
 	return r;
 }
@@ -367,7 +370,15 @@ static int uxsock_trigger(struct client *c, void *trigger_data)
 		return r;
 	}
 
-	r = parse_cmd(c, vecs, uxsock_timeout / 1000);
+	r = parse_cmd(c);
+
+	if (r == 0 && c->handler)
+		r = execute_handler(c, vecs, uxsock_timeout / 1000);
+
+	if (c->cmdvec) {
+		free_keys(c->cmdvec);
+		c->cmdvec = NULL;
+	}
 
 	if (r > 0) {
 		if (r == ETIMEDOUT)
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 28/48] multipathd: uxlsnr: use parser to determine non-root commands

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Rather than using a separate poor-man's parser for checking root
commands, use the real parser. It will return "LIST" as first verb
for the read-only commands that non-root users may execute.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index cfff0ae..ff9604f 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -362,16 +362,15 @@ static int uxsock_trigger(struct client *c, void *trigger_data)
 
 	vecs = (struct vectors *)trigger_data;
 
-
-	if (!c->is_root &&
-	    (strncmp(c->cmd, "list", strlen("list")) != 0) &&
-	    (strncmp(c->cmd, "show", strlen("show")) != 0)) {
-		append_strbuf_str(&c->reply, "permission deny: need to be root");
-		return r;
-	}
-
 	r = parse_cmd(c);
 
+	if (r == 0 && c->cmdvec && VECTOR_SIZE(c->cmdvec) > 0) {
+		struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
+
+		if (!c->is_root && kw->code != LIST)
+			r = EPERM;
+	}
+
 	if (r == 0 && c->handler)
 		r = execute_handler(c, vecs, uxsock_timeout / 1000);
 
@@ -381,10 +380,18 @@ static int uxsock_trigger(struct client *c, void *trigger_data)
 	}
 
 	if (r > 0) {
-		if (r == ETIMEDOUT)
+		switch(r) {
+		case ETIMEDOUT:
 			append_strbuf_str(&c->reply, "timeout\n");
-		else
+			break;
+		case EPERM:
+			append_strbuf_str(&c->reply,
+					  "permission deny: need to be root\n");
+			break;
+		default:
 			append_strbuf_str(&c->reply, "fail\n");
+			break;
+		}
 	}
 	else if (!r && get_strbuf_len(&c->reply) == 0) {
 		append_strbuf_str(&c->reply, "ok\n");
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[mwilck]

From: Martin Wilck <mwilck@suse.com>

This patch sets up the bulk of the state machine. client_state_machine()
is called in a loop, proceeding from state to state until it needs
to poll for input or wait for a lock, in which case it returns
STM_BREAK.

While doing this, switch to negative error codes for the functions
in uxlsnr.c (e.g. parse_cmd()). Positive return codes are reserved
for the cli_handler functions themselves. This way we can clearly
distinguish the error source, and avoid confusion and misleading
error messages. No cli_handler returns negative values.

Note: with this patch applied, clients may hang and time out if
the handler fails to acquire the vecs lock. This will be fixed in the
follow-up patch "multipathd: uxlsnr: add idle notification".

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 160 ++++++++++++++++++++++++--------------------
 1 file changed, 89 insertions(+), 71 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index ff9604f..87134d5 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -299,22 +299,13 @@ static int parse_cmd(struct client *c)
 
 	r = get_cmdvec(c->cmd, &c->cmdvec);
 
-	if (r) {
-		genhelp_handler(c->cmd, r, &c->reply);
-		if (get_strbuf_len(&c->reply) == 0)
-			return EINVAL;
-		return 0;
-	}
+	if (r)
+		return -r;
 
 	c->handler = find_handler_for_cmdvec(c->cmdvec);
 
-	if (!c->handler || !c->handler->fn) {
-		genhelp_handler(c->cmd, EINVAL, &c->reply);
-		if (get_strbuf_len(&c->reply) == 0)
-			r = EINVAL;
-		else
-			r = 0;
-	}
+	if (!c->handler || !c->handler->fn)
+		return -EINVAL;
 
 	return r;
 }
@@ -325,7 +316,7 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
 	struct timespec tmo;
 
 	if (!c->handler)
-		return EINVAL;
+		return -EINVAL;
 
 	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
 		tmo.tv_sec += timeout;
@@ -355,50 +346,30 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
 	return r;
 }
 
-static int uxsock_trigger(struct client *c, void *trigger_data)
+void default_reply(struct client *c, int r)
 {
-	struct vectors * vecs;
-	int r = 1;
-
-	vecs = (struct vectors *)trigger_data;
-
-	r = parse_cmd(c);
-
-	if (r == 0 && c->cmdvec && VECTOR_SIZE(c->cmdvec) > 0) {
-		struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
-
-		if (!c->is_root && kw->code != LIST)
-			r = EPERM;
-	}
-
-	if (r == 0 && c->handler)
-		r = execute_handler(c, vecs, uxsock_timeout / 1000);
-
-	if (c->cmdvec) {
-		free_keys(c->cmdvec);
-		c->cmdvec = NULL;
-	}
-
-	if (r > 0) {
-		switch(r) {
-		case ETIMEDOUT:
-			append_strbuf_str(&c->reply, "timeout\n");
-			break;
-		case EPERM:
-			append_strbuf_str(&c->reply,
-					  "permission deny: need to be root\n");
-			break;
-		default:
-			append_strbuf_str(&c->reply, "fail\n");
-			break;
-		}
-	}
-	else if (!r && get_strbuf_len(&c->reply) == 0) {
+	switch(r) {
+	case -EINVAL:
+	case -ESRCH:
+	case -ENOMEM:
+		/* return codes from get_cmdvec() */
+		genhelp_handler(c->cmd, -r, &c->reply);
+		break;
+	case -EPERM:
+		append_strbuf_str(&c->reply,
+				  "permission deny: need to be root\n");
+		break;
+	case -ETIMEDOUT:
+		append_strbuf_str(&c->reply, "timeout\n");
+		break;
+	case 0:
 		append_strbuf_str(&c->reply, "ok\n");
-		r = 0;
+		break;
+	default:
+		/* cli_handler functions return 1 on unspecified error */
+		append_strbuf_str(&c->reply, "fail\n");
+		break;
 	}
-	/* else if (r < 0) leave *reply alone */
-	return r;
 }
 
 static void set_client_state(struct client *c, int state)
@@ -409,6 +380,7 @@ static void set_client_state(struct client *c, int state)
 		reset_strbuf(&c->reply);
 		memset(c->cmd, '\0', sizeof(c->cmd));
 		c->expires = ts_zero;
+		c->error = 0;
 		/* fallthrough */
 	case CLT_SEND:
 		/* reuse these fields for next data transfer */
@@ -420,11 +392,20 @@ static void set_client_state(struct client *c, int state)
 	c->state = state;
 }
 
-static void handle_client(struct client *c, void *trigger_data)
+enum {
+	STM_CONT,
+	STM_BREAK,
+};
+
+static int client_state_machine(struct client *c, struct vectors *vecs)
 {
 	ssize_t n;
+	const char *buf;
 
-	switch (c->state) {
+	condlog(4, "%s: cli[%d] state=%d cmd=\"%s\" repl \"%s\"", __func__,
+		c->fd, c->state, c->cmd, get_strbuf_str(&c->reply));
+
+        switch (c->state) {
 	case CLT_RECV:
 		if (c->cmd_len == 0) {
 			/*
@@ -449,31 +430,59 @@ static void handle_client(struct client *c, void *trigger_data)
 				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
 			}
 			/* poll for data */
-			return;
+			return STM_BREAK;
 		} else if (c->len < c->cmd_len) {
 			n = recv(c->fd, c->cmd + c->len, c->cmd_len - c->len, 0);
 			if (n <= 0 && errno != EINTR && errno != EAGAIN) {
 				condlog(1, "%s: cli[%d]: error in recv: %m",
 					__func__, c->fd);
 				c->error = -ECONNRESET;
-				return;
+				return STM_BREAK;
 			}
 			c->len += n;
 			if (c->len < c->cmd_len)
 				/* continue polling */
-				return;
-			set_client_state(c, CLT_PARSE);
+				return STM_BREAK;
 		}
-		break;
-	default:
-		break;
-	}
+		condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
+		set_client_state(c, CLT_PARSE);
+		return STM_CONT;
 
-	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
-	uxsock_trigger(c, trigger_data);
+	case CLT_PARSE:
+		c->error = parse_cmd(c);
+		if (!c->error) {
+			/* Permission check */
+			struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
 
-	if (get_strbuf_len(&c->reply) > 0) {
-		const char *buf = get_strbuf_str(&c->reply);
+			if (!c->is_root && kw->code != LIST) {
+				c->error = -EPERM;
+				condlog(0, "%s: cli[%d]: unauthorized cmd \"%s\"",
+					__func__, c->fd, c->cmd);
+			}
+		}
+		if (c->error)
+			set_client_state(c, CLT_SEND);
+		else
+			set_client_state(c, CLT_WORK);
+		return STM_CONT;
+
+	case CLT_WAIT_LOCK:
+		/* tbd */
+		set_client_state(c, CLT_WORK);
+		return STM_CONT;
+
+	case CLT_WORK:
+		c->error = execute_handler(c, vecs, uxsock_timeout / 1000);
+		free_keys(c->cmdvec);
+		c->cmdvec = NULL;
+		set_client_state(c, CLT_SEND);
+		return STM_CONT;
+
+	case CLT_SEND:
+		if (get_strbuf_len(&c->reply) == 0)
+			default_reply(c, c->error);
+
+		buf = get_strbuf_str(&c->reply);
 
 		if (send_packet(c->fd, buf) != 0)
 			dead_client(c);
@@ -481,9 +490,18 @@ static void handle_client(struct client *c, void *trigger_data)
 			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
 				get_strbuf_len(&c->reply) + 1);
 		reset_strbuf(&c->reply);
-	}
 
-	set_client_state(c, CLT_RECV);
+		set_client_state(c, CLT_RECV);
+		return STM_BREAK;
+
+	default:
+		return STM_BREAK;
+	}
+}
+
+static void handle_client(struct client *c, struct vectors *vecs)
+{
+	while (client_state_machine(c, vecs) == STM_CONT);
 }
 
 /*
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 30/48] multipathd: uxlsnr: add idle notification

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The previous patches added the state machine and the timeout handling,
but there was no wakeup mechanism for the uxlsnr for cases where
client connections were waiting for the vecs lock.

This patch uses the previously introduced wakeup mechanism of
struct mutex_lock for this purpose. Processes which unlock the
"global" vecs lock send an event in an eventfd which the uxlsnr
loop is polling for.

As we are now woken up for servicing client handlers that don't
wait for input but for the lock, we need to set up the pollfds
differently, and iterate over all clients when handling events,
not only over the ones that are receiving. The hangup handling
is changed, too. We have to look at every client, even if one has
hung up. Note that I don't take client_lock for the loop in
uxsock_listen(), it's not necessary and will be removed elsewhere
in a follow-up patch.

With this in place, the lock need not be taken in execute_handler()
any more. The uxlsnr only ever calls trylock() on the vecs lock,
avoiding any waiting for other threads to finish.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 200 ++++++++++++++++++++++++++++----------------
 1 file changed, 129 insertions(+), 71 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 87134d5..bf9780d 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -24,6 +24,7 @@
 #include <signal.h>
 #include <stdbool.h>
 #include <sys/inotify.h>
+#include <sys/eventfd.h>
 #include "checkers.h"
 #include "memory.h"
 #include "debug.h"
@@ -70,6 +71,7 @@ struct client {
 enum {
 	POLLFD_UX = 0,
 	POLLFD_NOTIFY,
+	POLLFD_IDLE,
 	POLLFDS_BASE,
 };
 
@@ -90,6 +92,7 @@ static LIST_HEAD(clients);
 static pthread_mutex_t client_lock = PTHREAD_MUTEX_INITIALIZER;
 static struct pollfd *polls;
 static int notify_fd = -1;
+static int idle_fd = -1;
 static char *watch_config_dir;
 
 static bool _socket_client_is_root(int fd)
@@ -187,6 +190,17 @@ void uxsock_cleanup(void *arg)
 	free_polls();
 }
 
+void wakeup_cleanup(void *arg)
+{
+	struct mutex_lock *lck = arg;
+	int fd = idle_fd;
+
+	idle_fd = -1;
+	set_wakeup_fn(lck, NULL);
+	if (fd != -1)
+		close(fd);
+}
+
 struct watch_descriptors {
 	int conf_wd;
 	int dir_wd;
@@ -293,6 +307,18 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 
 static const struct timespec ts_zero = { .tv_sec = 0, };
 
+/* call with clients lock held */
+static bool __need_vecs_lock(void)
+{
+	struct client *c;
+
+	list_for_each_entry(c, &clients, node) {
+		if (c->state == CLT_WAIT_LOCK)
+			return true;
+	}
+	return false;
+}
+
 static int parse_cmd(struct client *c)
 {
 	int r;
@@ -310,40 +336,31 @@ static int parse_cmd(struct client *c)
 	return r;
 }
 
-static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
+static int execute_handler(struct client *c, struct vectors *vecs)
 {
-	int r;
-	struct timespec tmo;
 
-	if (!c->handler)
+	if (!c->handler || !c->handler->fn)
 		return -EINVAL;
 
-	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
-		tmo.tv_sec += timeout;
-	} else {
-		tmo.tv_sec = 0;
-	}
+	return c->handler->fn(c->cmdvec, &c->reply, vecs);
+}
 
-	if (c->handler->locked) {
-		int locked = 0;
+static void wakeup_listener(void)
+{
+	uint64_t one = 1;
 
-		pthread_cleanup_push(cleanup_lock, &vecs->lock);
-		if (tmo.tv_sec) {
-			r = timedlock(&vecs->lock, &tmo);
-		} else {
-			lock(&vecs->lock);
-			r = 0;
-		}
-		if (r == 0) {
-			locked = 1;
-			pthread_testcancel();
-			r = c->handler->fn(c->cmdvec, &c->reply, vecs);
-		}
-		pthread_cleanup_pop(locked);
-	} else
-		r = c->handler->fn(c->cmdvec, &c->reply, vecs);
+	if (idle_fd != -1 &&
+	    write(idle_fd, &one, sizeof(one)) != sizeof(one))
+		condlog(1, "%s: failed", __func__);
+}
 
-	return r;
+static void drain_idle_fd(int fd)
+{
+	uint64_t val;
+	int rc;
+
+	rc = read(fd, &val, sizeof(val));
+	condlog(4, "%s: %d, %"PRIu64, __func__, rc, val);
 }
 
 void default_reply(struct client *c, int r)
@@ -397,16 +414,19 @@ enum {
 	STM_BREAK,
 };
 
-static int client_state_machine(struct client *c, struct vectors *vecs)
+static int client_state_machine(struct client *c, struct vectors *vecs,
+				short revents)
 {
 	ssize_t n;
 	const char *buf;
 
-	condlog(4, "%s: cli[%d] state=%d cmd=\"%s\" repl \"%s\"", __func__,
-		c->fd, c->state, c->cmd, get_strbuf_str(&c->reply));
+	condlog(4, "%s: cli[%d] poll=%x state=%d cmd=\"%s\" repl \"%s\"", __func__,
+		c->fd, revents, c->state, c->cmd, get_strbuf_str(&c->reply));
 
         switch (c->state) {
 	case CLT_RECV:
+		if (!(revents & POLLIN))
+			return STM_BREAK;
 		if (c->cmd_len == 0) {
 			/*
 			 * We got POLLIN; assume that at least the length can
@@ -462,17 +482,30 @@ static int client_state_machine(struct client *c, struct vectors *vecs)
 		}
 		if (c->error)
 			set_client_state(c, CLT_SEND);
+		else if (c->handler->locked)
+			set_client_state(c, CLT_WAIT_LOCK);
 		else
 			set_client_state(c, CLT_WORK);
 		return STM_CONT;
 
 	case CLT_WAIT_LOCK:
-		/* tbd */
-		set_client_state(c, CLT_WORK);
-		return STM_CONT;
+                if (trylock(&vecs->lock) == 0) {
+			/* don't use cleanup_lock(), lest we wakeup ourselves */
+			pthread_cleanup_push_cast(__unlock, &vecs->lock);
+			c->error = execute_handler(c, vecs);
+			pthread_cleanup_pop(1);
+			condlog(4, "%s: cli[%d] grabbed lock", __func__, c->fd);
+			free_keys(c->cmdvec);
+			c->cmdvec = NULL;
+			set_client_state(c, CLT_SEND);
+			return STM_CONT;
+		} else {
+			condlog(4, "%s: cli[%d] waiting for lock", __func__, c->fd);
+			return STM_BREAK;
+		}
 
 	case CLT_WORK:
-		c->error = execute_handler(c, vecs, uxsock_timeout / 1000);
+		c->error = execute_handler(c, vecs);
 		free_keys(c->cmdvec);
 		c->cmdvec = NULL;
 		set_client_state(c, CLT_SEND);
@@ -499,9 +532,14 @@ static int client_state_machine(struct client *c, struct vectors *vecs)
 	}
 }
 
-static void handle_client(struct client *c, struct vectors *vecs)
+static void handle_client(struct client *c, struct vectors *vecs, short revents)
 {
-	while (client_state_machine(c, vecs) == STM_CONT);
+	if (revents & (POLLHUP|POLLERR)) {
+		c->error = -ECONNRESET;
+		return;
+	}
+
+        while (client_state_machine(c, vecs, revents) == STM_CONT);
 }
 
 /*
@@ -514,6 +552,8 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 	/* conf->sequence_nr will be 1 when uxsock_listen is first called */
 	unsigned int sequence_nr = 0;
 	struct watch_descriptors wds = { .conf_wd = -1, .dir_wd = -1 };
+	bool need_lock = false;
+	struct vectors *vecs = trigger_data;
 
 	condlog(3, "uxsock: startup listener");
 	polls = MALLOC(max_pfds * sizeof(*polls));
@@ -524,6 +564,14 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 	notify_fd = inotify_init1(IN_NONBLOCK);
 	if (notify_fd == -1) /* it's fine if notifications fail */
 		condlog(3, "failed to start up configuration notifications");
+
+	pthread_cleanup_push(wakeup_cleanup, &vecs->lock);
+	idle_fd = eventfd(0, EFD_NONBLOCK|EFD_CLOEXEC);
+	if (idle_fd == -1)
+		condlog(1, "failed to create idle fd");
+	else
+		set_wakeup_fn(&vecs->lock, wakeup_listener);
+
 	sigfillset(&mask);
 	sigdelset(&mask, SIGINT);
 	sigdelset(&mask, SIGTERM);
@@ -575,16 +623,30 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 		else
 			polls[POLLFD_NOTIFY].events = POLLIN;
 
+		need_lock = __need_vecs_lock();
+		polls[POLLFD_IDLE].fd = idle_fd;
+		if (need_lock)
+			polls[POLLFD_IDLE].events = POLLIN;
+		else
+			polls[POLLFD_IDLE].events = 0;
+
 		/* setup the clients */
-		i = POLLFDS_BASE;
-		list_for_each_entry(c, &clients, node) {
-			polls[i].fd = c->fd;
-			polls[i].events = POLLIN;
-			i++;
-			if (i >= max_pfds)
-				break;
-		}
-		n_pfds = i;
+                i = POLLFDS_BASE;
+                list_for_each_entry(c, &clients, node) {
+                        switch(c->state) {
+                        case CLT_RECV:
+                                polls[i].events = POLLIN;
+                                break;
+                        default:
+				/* don't poll for this client */
+                                continue;
+                        }
+                        polls[i].fd = c->fd;
+                        i++;
+                        if (i >= max_pfds)
+                                break;
+                }
+                n_pfds = i;
 		pthread_cleanup_pop(1);
 
 		/* most of our life is spent in this call */
@@ -607,33 +669,28 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 			handle_signals(true);
 			continue;
 		}
+		if (polls[POLLFD_IDLE].fd != -1 &&
+		    polls[POLLFD_IDLE].revents & POLLIN)
+			drain_idle_fd(idle_fd);
 
-		/* see if a client wants to speak to us */
-		for (i = POLLFDS_BASE; i < n_pfds; i++) {
-			if (polls[i].revents & (POLLIN|POLLHUP|POLLERR)) {
-				c = NULL;
-				pthread_mutex_lock(&client_lock);
-				list_for_each_entry(tmp, &clients, node) {
-					if (tmp->fd == polls[i].fd) {
-						c = tmp;
-						break;
-					}
-				}
-				pthread_mutex_unlock(&client_lock);
-				if (!c) {
-					condlog(4, "cli%d: new fd %d",
-						i, polls[i].fd);
-					continue;
-				}
-				if (polls[i].revents & (POLLHUP|POLLERR)) {
-					condlog(4, "cli[%d]: Disconnected",
-						c->fd);
-					dead_client(c);
-					continue;
-				}
-				handle_client(c, trigger_data);
-				if (c->error == -ECONNRESET)
-					dead_client(c);
+		/* see if a client needs handling */
+		list_for_each_entry_safe(c, tmp, &clients, node) {
+			short revents = 0;
+
+			for (i = POLLFDS_BASE; i < n_pfds; i++) {
+                                if (polls[i].fd == c->fd) {
+                                        revents = polls[i].revents;
+                                        break;
+                                }
+                        }
+
+			handle_client(c, trigger_data, revents);
+
+			if (c->error == -ECONNRESET) {
+				condlog(4, "cli[%d]: disconnected", c->fd);
+				dead_client(c);
+				if (i < n_pfds)
+					polls[i].fd = -1;
 			}
 		}
 		/* see if we got a non-fatal signal */
@@ -649,5 +706,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 			handle_inotify(notify_fd, &wds);
 	}
 
+	pthread_cleanup_pop(1);
 	return NULL;
 }
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 31/48] multipathd: uxlsnr: add timeout handling

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Our ppoll() call needs to wake up when a client request times out.
This logic can be added by determining the first client that's about
to time out. The logic in handle_client() will then cause a timeout
reply to be sent to the client. This is more client-friendly
as the client timing out without receiving a reply.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 57 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 5 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index bf9780d..45fe7b5 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -306,6 +306,35 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 }
 
 static const struct timespec ts_zero = { .tv_sec = 0, };
+static const struct timespec ts_max = { .tv_sec = LONG_MAX, .tv_nsec = 999999999 };
+
+/* call with clients lock held */
+static struct timespec *__get_soonest_timeout(struct timespec *ts)
+{
+	struct timespec ts_min = ts_max, now;
+	bool any = false;
+	struct client *c;
+
+	list_for_each_entry(c, &clients, node) {
+		if (timespeccmp(&c->expires, &ts_zero) != 0 &&
+		    timespeccmp(&c->expires, &ts_min) < 0) {
+			ts_min = c->expires;
+			any = true;
+		}
+	}
+
+	if (!any)
+		return NULL;
+
+	get_monotonic_time(&now);
+	timespecsub(&ts_min, &now, ts);
+	if (timespeccmp(ts, &ts_zero) < 0)
+		*ts = ts_zero;
+
+	condlog(4, "%s: next client expires in %ld.%03lds", __func__,
+		(long)ts->tv_sec, ts->tv_nsec / 1000000);
+	return ts;
+}
 
 /* call with clients lock held */
 static bool __need_vecs_lock(void)
@@ -532,6 +561,24 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
 	}
 }
 
+static void check_timeout(struct client *c)
+{
+	struct timespec now;
+
+	if (timespeccmp(&c->expires, &ts_zero) == 0)
+		return;
+
+	get_monotonic_time(&now);
+	if (timespeccmp(&c->expires, &now) > 0)
+		return;
+
+	condlog(2, "%s: cli[%d]: timed out at %ld.%03ld", __func__,
+		c->fd, (long)c->expires.tv_sec, c->expires.tv_nsec / 1000000);
+
+	c->error = -ETIMEDOUT;
+	set_client_state(c, CLT_SEND);
+}
+
 static void handle_client(struct client *c, struct vectors *vecs, short revents)
 {
 	if (revents & (POLLHUP|POLLERR)) {
@@ -539,6 +586,7 @@ static void handle_client(struct client *c, struct vectors *vecs, short revents)
 		return;
 	}
 
+	check_timeout(c);
         while (client_state_machine(c, vecs, revents) == STM_CONT);
 }
 
@@ -580,6 +628,7 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 	while (1) {
 		struct client *c, *tmp;
 		int i, n_pfds, poll_count, num_clients;
+		struct timespec __timeout, *timeout;
 
 		/* setup for a poll */
 		pthread_mutex_lock(&client_lock);
@@ -647,10 +696,12 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
                                 break;
                 }
                 n_pfds = i;
+		timeout = __get_soonest_timeout(&__timeout);
+
 		pthread_cleanup_pop(1);
 
 		/* most of our life is spent in this call */
-		poll_count = ppoll(polls, n_pfds, NULL, &mask);
+		poll_count = ppoll(polls, n_pfds, timeout, &mask);
 
 		handle_signals(false);
 		if (poll_count == -1) {
@@ -665,10 +716,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 			break;
 		}
 
-		if (poll_count == 0) {
-			handle_signals(true);
-			continue;
-		}
 		if (polls[POLLFD_IDLE].fd != -1 &&
 		    polls[POLLFD_IDLE].revents & POLLIN)
 			drain_idle_fd(idle_fd);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 32/48] multipathd: uxlsnr: use poll loop for sending, too

[mwilck]

From: Martin Wilck <mwilck@suse.com>

send_packet() may busy-loop. By polling for POLLOUT, we can
avoid that, even if it's very unlikely in practice.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 45fe7b5..6213454 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -447,7 +447,6 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
 				short revents)
 {
 	ssize_t n;
-	const char *buf;
 
 	condlog(4, "%s: cli[%d] poll=%x state=%d cmd=\"%s\" repl \"%s\"", __func__,
 		c->fd, revents, c->state, c->cmd, get_strbuf_str(&c->reply));
@@ -544,16 +543,31 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
 		if (get_strbuf_len(&c->reply) == 0)
 			default_reply(c, c->error);
 
-		buf = get_strbuf_str(&c->reply);
+		if (c->cmd_len == 0) {
+			size_t len = get_strbuf_len(&c->reply) + 1;
 
-		if (send_packet(c->fd, buf) != 0)
-			dead_client(c);
-		else
-			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
-				get_strbuf_len(&c->reply) + 1);
-		reset_strbuf(&c->reply);
+			if (send(c->fd, &len, sizeof(len), MSG_NOSIGNAL)
+			    != sizeof(len))
+				c->error = -ECONNRESET;
+			c->cmd_len = len;
+			return STM_BREAK;
+		}
 
-		set_client_state(c, CLT_RECV);
+		if (c->len < c->cmd_len) {
+			const char *buf = get_strbuf_str(&c->reply);
+
+			n = send(c->fd, buf + c->len, c->cmd_len, MSG_NOSIGNAL);
+			if (n == -1) {
+				if (!(errno == EAGAIN || errno == EINTR))
+					c->error = -ECONNRESET;
+			} else
+				c->len += n;
+		}
+
+                if (c->len >= c->cmd_len) {
+			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd, c->cmd_len);
+			set_client_state(c, CLT_RECV);
+		}
 		return STM_BREAK;
 
 	default:
@@ -686,6 +700,9 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
                         case CLT_RECV:
                                 polls[i].events = POLLIN;
                                 break;
+			case CLT_SEND:
+				polls[i].events = POLLOUT;
+				break;
                         default:
 				/* don't poll for this client */
                                 continue;
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 33/48] multipathd: uxlsnr: drop client_lock

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The list of clients is never changed anywhere except in
uxsock_listen(). No need to lock.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 33 ++++++---------------------------
 1 file changed, 6 insertions(+), 27 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 6213454..24db377 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -89,7 +89,6 @@ enum {
 static __attribute__((unused)) char ___a[-(MIN_POLLS <= 0)];
 
 static LIST_HEAD(clients);
-static pthread_mutex_t client_lock = PTHREAD_MUTEX_INITIALIZER;
 static struct pollfd *polls;
 static int notify_fd = -1;
 static int idle_fd = -1;
@@ -136,15 +135,13 @@ static void new_client(int ux_sock)
 	c->is_root = _socket_client_is_root(c->fd);
 
 	/* put it in our linked list */
-	pthread_mutex_lock(&client_lock);
 	list_add_tail(&c->node, &clients);
-	pthread_mutex_unlock(&client_lock);
 }
 
 /*
  * kill off a dead client
  */
-static void _dead_client(struct client *c)
+static void dead_client(struct client *c)
 {
 	int fd = c->fd;
 	list_del_init(&c->node);
@@ -156,14 +153,6 @@ static void _dead_client(struct client *c)
 	close(fd);
 }
 
-static void dead_client(struct client *c)
-{
-	pthread_cleanup_push(cleanup_mutex, &client_lock);
-	pthread_mutex_lock(&client_lock);
-	_dead_client(c);
-	pthread_cleanup_pop(1);
-}
-
 static void free_polls (void)
 {
 	if (polls)
@@ -180,11 +169,9 @@ void uxsock_cleanup(void *arg)
 	close(notify_fd);
 	free(watch_config_dir);
 
-	pthread_mutex_lock(&client_lock);
 	list_for_each_entry_safe(client_loop, client_tmp, &clients, node) {
-		_dead_client(client_loop);
+		dead_client(client_loop);
 	}
-	pthread_mutex_unlock(&client_lock);
 
 	cli_exit();
 	free_polls();
@@ -308,8 +295,7 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
 static const struct timespec ts_zero = { .tv_sec = 0, };
 static const struct timespec ts_max = { .tv_sec = LONG_MAX, .tv_nsec = 999999999 };
 
-/* call with clients lock held */
-static struct timespec *__get_soonest_timeout(struct timespec *ts)
+static struct timespec *get_soonest_timeout(struct timespec *ts)
 {
 	struct timespec ts_min = ts_max, now;
 	bool any = false;
@@ -336,8 +322,7 @@ static struct timespec *__get_soonest_timeout(struct timespec *ts)
 	return ts;
 }
 
-/* call with clients lock held */
-static bool __need_vecs_lock(void)
+static bool need_vecs_lock(void)
 {
 	struct client *c;
 
@@ -614,7 +599,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 	/* conf->sequence_nr will be 1 when uxsock_listen is first called */
 	unsigned int sequence_nr = 0;
 	struct watch_descriptors wds = { .conf_wd = -1, .dir_wd = -1 };
-	bool need_lock = false;
 	struct vectors *vecs = trigger_data;
 
 	condlog(3, "uxsock: startup listener");
@@ -645,8 +629,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 		struct timespec __timeout, *timeout;
 
 		/* setup for a poll */
-		pthread_mutex_lock(&client_lock);
-		pthread_cleanup_push(cleanup_mutex, &client_lock);
 		num_clients = 0;
 		list_for_each_entry(c, &clients, node) {
 			num_clients++;
@@ -686,9 +668,8 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
 		else
 			polls[POLLFD_NOTIFY].events = POLLIN;
 
-		need_lock = __need_vecs_lock();
 		polls[POLLFD_IDLE].fd = idle_fd;
-		if (need_lock)
+		if (need_vecs_lock())
 			polls[POLLFD_IDLE].events = POLLIN;
 		else
 			polls[POLLFD_IDLE].events = 0;
@@ -713,9 +694,7 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
                                 break;
                 }
                 n_pfds = i;
-		timeout = __get_soonest_timeout(&__timeout);
-
-		pthread_cleanup_pop(1);
+		timeout = get_soonest_timeout(&__timeout);
 
 		/* most of our life is spent in this call */
 		poll_count = ppoll(polls, n_pfds, timeout, &mask);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 34/48] multipathd: uxclt: allow client mode for non-root, too

[mwilck]

From: Martin Wilck <mwilck@suse.com>

The server checks for root permissions anyway. "multipathd -k"
should work for ordinary users as long as no priviledged commands
are executed.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/main.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index dbe6dc7..7c3e9b2 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -3348,11 +3348,6 @@ main (int argc, char *argv[])
 
 	logsink = LOGSINK_SYSLOG;
 
-	if (getuid() != 0) {
-		fprintf(stderr, "need to be root\n");
-		exit(1);
-	}
-
 	/* make sure we don't lock any path */
 	if (chdir("/") < 0)
 		fprintf(stderr, "can't chdir to root directory : %s\n",
@@ -3439,6 +3434,11 @@ main (int argc, char *argv[])
 		return err;
 	}
 
+	if (getuid() != 0) {
+		fprintf(stderr, "need to be root\n");
+		exit(1);
+	}
+
 	if (foreground) {
 		if (!isatty(fileno(stdout)))
 			setbuf(stdout, NULL);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 35/48] multipathd: uxlsnr: use recv() for command length

[mwilck]

From: Martin Wilck <mwilck@suse.com>

If the peer uses libmpathcmd, we can be certain that the first
8 bytes are being sent in a single chunk of data. It's overkill
to try and receive the command length byte-by-byte.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/uxlsnr.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 24db377..6355279 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -441,6 +441,7 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
 		if (!(revents & POLLIN))
 			return STM_BREAK;
 		if (c->cmd_len == 0) {
+			size_t len;
 			/*
 			 * We got POLLIN; assume that at least the length can
 			 * be read immediately.
@@ -449,17 +450,17 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
 			c->expires.tv_sec += uxsock_timeout / 1000;
 			c->expires.tv_nsec += (uxsock_timeout % 1000) * 1000000;
 			normalize_timespec(&c->expires);
-			n = mpath_recv_reply_len(c->fd, 0);
-			if (n == -1) {
-				condlog(1, "%s: cli[%d]: failed to receive reply len",
-					__func__, c->fd);
-				c->error = -ECONNRESET;
-			} else if (n > _MAX_CMD_LEN) {
-				condlog(1, "%s: cli[%d]: overlong command (%zd bytes)",
+			n = recv(c->fd, &len, sizeof(len), 0);
+			if (n < (ssize_t)sizeof(len)) {
+				condlog(1, "%s: cli[%d]: failed to receive reply len: %zd",
 					__func__, c->fd, n);
 				c->error = -ECONNRESET;
+			} else if (len <= 0 || len > _MAX_CMD_LEN) {
+				condlog(1, "%s: cli[%d]: invalid command length (%zu bytes)",
+					__func__, c->fd, len);
+				c->error = -ECONNRESET;
 			} else {
-				c->cmd_len = n;
+				c->cmd_len = len;
 				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
 			}
 			/* poll for data */
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 36/48] multipathd: move delayed_reconfig out of struct config

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

delayed_reconfig was inside the config struct, but it wasn't updated
during an RCU write section, so there's no synchronization on it.
Instead, pull it out of the config structure, and use the config_lock
to synchronize it.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmpathpersist/libmpathpersist.version | 12 +++----
 libmultipath/config.h                   |  1 -
 libmultipath/libmultipath.version       |  9 ++---
 multipathd/main.c                       | 45 ++++++++++++++++---------
 4 files changed, 37 insertions(+), 30 deletions(-)

diff --git a/libmpathpersist/libmpathpersist.version b/libmpathpersist/libmpathpersist.version
index e074813..b52b750 100644
--- a/libmpathpersist/libmpathpersist.version
+++ b/libmpathpersist/libmpathpersist.version
@@ -10,7 +10,7 @@
  *
  * See libmultipath.version for general policy about version numbers.
  */
-LIBMPATHPERSIST_1.0.0 {
+LIBMPATHPERSIST_2.0.0 {
 global:
 
 	__mpath_persistent_reserve_in;
@@ -28,11 +28,9 @@ global:
 	prout_do_scsi_ioctl;
 	update_map_pr;
 
-local: *;
-};
-
-LIBMPATHPERSIST_1.1.0 {
-global:
+	/* added int 1.1.0 */
 	libmpathpersist_init;
 	libmpathpersist_exit;
-} LIBMPATHPERSIST_1.0.0;
+
+local: *;
+};
diff --git a/libmultipath/config.h b/libmultipath/config.h
index 933fe0d..c73389b 100644
--- a/libmultipath/config.h
+++ b/libmultipath/config.h
@@ -178,7 +178,6 @@ struct config {
 	int strict_timing;
 	int retrigger_tries;
 	int retrigger_delay;
-	int delayed_reconfig;
 	int uev_wait_timeout;
 	int skip_kpartx;
 	int remove_retries;
diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index 2fb2547..4f6fa1d 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -31,7 +31,7 @@
  *   The new version inherits the previous ones.
  */
 
-LIBMULTIPATH_10.0.0 {
+LIBMULTIPATH_11.0.0 {
 global:
 	/* symbols referenced by multipath and multipathd */
 	add_foreign;
@@ -290,11 +290,8 @@ global:
 	/* added in 9.2.0 */
 	set_wakeup_fn;
 
+	/* added in 10.1.0 */
+	__snprint_config;
 local:
 	*;
 };
-
-LIBMULTIPATH_10.1.0 {
-global:
-	__snprint_config;
-} LIBMULTIPATH_10.0.0;
diff --git a/multipathd/main.c b/multipathd/main.c
index 7c3e9b2..57c572d 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -127,6 +127,8 @@ static int poll_dmevents = 1;
 #endif
 /* Don't access this variable without holding config_lock */
 static volatile enum daemon_status running_state = DAEMON_INIT;
+/* Don't access this variable without holding config_lock */
+static bool __delayed_reconfig;
 pid_t daemon_pid;
 static pthread_mutex_t config_lock = PTHREAD_MUTEX_INITIALIZER;
 static pthread_cond_t config_cond;
@@ -150,6 +152,23 @@ int should_exit(void)
 	return get_running_state() == DAEMON_SHUTDOWN;
 }
 
+static bool get_delayed_reconfig(void)
+{
+	bool val;
+
+	pthread_mutex_lock(&config_lock);
+	val = __delayed_reconfig;
+	pthread_mutex_unlock(&config_lock);
+	return val;
+}
+
+static void set_delayed_reconfig(bool val)
+{
+	pthread_mutex_lock(&config_lock);
+	__delayed_reconfig = val;
+	pthread_mutex_unlock(&config_lock);
+}
+
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -297,8 +316,10 @@ static void __post_config_state(enum daemon_status state)
 			old_state = DAEMON_IDLE;
 			state = DAEMON_CONFIGURE;
 		}
-		if (reconfigure_pending && state == DAEMON_CONFIGURE)
+		if (state == DAEMON_CONFIGURE) {
 			reconfigure_pending = false;
+			__delayed_reconfig = false;
+		}
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
 		do_sd_notify(old_state, state);
@@ -765,7 +786,7 @@ int
 ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 {
 	struct multipath * mpp;
-	int delayed_reconfig, reassign_maps;
+	int reassign_maps;
 	struct config *conf;
 
 	if (dm_is_mpath(alias) != 1) {
@@ -784,12 +805,11 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 				return 1;
 		}
 		conf = get_multipath_config();
-		delayed_reconfig = conf->delayed_reconfig;
 		reassign_maps = conf->reassign_maps;
 		put_multipath_config(conf);
 		if (mpp->wait_for_udev) {
 			mpp->wait_for_udev = 0;
-			if (delayed_reconfig &&
+			if (get_delayed_reconfig() &&
 			    !need_to_delay_reconfig(vecs)) {
 				condlog(2, "reconfigure (delayed)");
 				schedule_reconfigure();
@@ -1791,8 +1811,7 @@ missing_uev_wait_tick(struct vectors *vecs)
 {
 	struct multipath * mpp;
 	unsigned int i;
-	int timed_out = 0, delayed_reconfig;
-	struct config *conf;
+	int timed_out = 0;
 
 	vector_foreach_slot (vecs->mpvec, mpp, i) {
 		if (mpp->wait_for_udev && --mpp->uev_wait_tick <= 0) {
@@ -1808,10 +1827,7 @@ missing_uev_wait_tick(struct vectors *vecs)
 		}
 	}
 
-	conf = get_multipath_config();
-	delayed_reconfig = conf->delayed_reconfig;
-	put_multipath_config(conf);
-	if (timed_out && delayed_reconfig &&
+	if (timed_out && get_delayed_reconfig() &&
 	    !need_to_delay_reconfig(vecs)) {
 		condlog(2, "reconfigure (delayed)");
 		schedule_reconfigure();
@@ -3257,13 +3273,10 @@ child (__attribute__((unused)) void *param)
 			pthread_cleanup_push(cleanup_lock, &vecs->lock);
 			lock(&vecs->lock);
 			pthread_testcancel();
-			if (!need_to_delay_reconfig(vecs)) {
+			if (!need_to_delay_reconfig(vecs))
 				reconfigure(vecs);
-			} else {
-				conf = get_multipath_config();
-				conf->delayed_reconfig = 1;
-				put_multipath_config(conf);
-			}
+			else
+				set_delayed_reconfig(true);
 			lock_cleanup_pop(vecs->lock);
 			post_config_state(DAEMON_IDLE);
 		}
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 37/48] multipathd: remove reconfigure from header file.

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

Only multipathd/main.c uses it.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/main.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/multipathd/main.h b/multipathd/main.h
index 2960a4d..7017380 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -39,7 +39,6 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 					    unsigned long ms);
 void schedule_reconfigure(void);
 int need_to_delay_reconfig (struct vectors *);
-int reconfigure (struct vectors *);
 int ev_add_path (struct path *, struct vectors *, int);
 int ev_remove_path (struct path *, struct vectors *, int);
 int ev_add_map (char *, const char *, struct vectors *);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 38/48] multipathd: pass in the type of reconfigure

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

schedule_reconfigure() now takes the type of reconfigure to do, and that
gets passed down to reconfigure(). If a full reconfigure has already
been requested but not started, weak reconfigure requests will be
upgraded. Currently cli_reconfigure() and the HUP signal request full
reconfigures, while the initial reconfigure, ev_add_map(), and
missing_uev_wait_tick() request weak reconfigures. A future patch will
enable users to control what kind of reconfigures happen.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/configure.c  |  2 +-
 multipathd/cli_handlers.c |  2 +-
 multipathd/main.c         | 55 +++++++++++++++++++++------------------
 multipathd/main.h         |  2 +-
 4 files changed, 32 insertions(+), 29 deletions(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index 7edb355..eb8ec1b 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -1098,7 +1098,7 @@ out:
  * FORCE_RELOAD_NONE: existing maps aren't touched at all
  * FORCE_RELOAD_YES: all maps are rebuilt from scratch and (re)loaded in DM
  * FORCE_RELOAD_WEAK: existing maps are compared to the current conf and only
- * reloaded in DM if there's a difference. This is useful during startup.
+ * reloaded in DM if there's a difference. This is normally sufficient.
  */
 int coalesce_paths (struct vectors *vecs, vector mpvec, char *refwwid,
 		    int force_reload, enum mpath_cmds cmd)
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index f59db3a..d165675 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -1012,7 +1012,7 @@ cli_reconfigure(void * v, struct strbuf *reply, void * data)
 {
 	condlog(2, "reconfigure (operator)");
 
-	schedule_reconfigure();
+	schedule_reconfigure(FORCE_RELOAD_YES);
 	return 0;
 }
 
diff --git a/multipathd/main.c b/multipathd/main.c
index 57c572d..1efbaab 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -162,13 +162,6 @@ static bool get_delayed_reconfig(void)
 	return val;
 }
 
-static void set_delayed_reconfig(bool val)
-{
-	pthread_mutex_lock(&config_lock);
-	__delayed_reconfig = val;
-	pthread_mutex_unlock(&config_lock);
-}
-
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -290,7 +283,18 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 }
 
 /* Don't access this variable without holding config_lock */
-static bool reconfigure_pending;
+static enum force_reload_types reconfigure_pending = FORCE_RELOAD_NONE;
+/* Only set while changing to DAEMON_CONFIGURE, and only access while
+ * reconfiguring or scheduling a delayed reconfig in DAEMON_CONFIGURE */
+static volatile enum force_reload_types reload_type = FORCE_RELOAD_NONE;
+
+static void enable_delayed_reconfig(void)
+{
+	pthread_mutex_lock(&config_lock);
+	reconfigure_pending = reload_type;
+	__delayed_reconfig = true;
+	pthread_mutex_unlock(&config_lock);
+}
 
 /* must be called with config_lock held */
 static void __post_config_state(enum daemon_status state)
@@ -305,7 +309,8 @@ static void __post_config_state(enum daemon_status state)
 		 * In either case, child() will see DAEMON_CONFIGURE
 		 * again and start another reconfigure cycle.
 		 */
-		if (reconfigure_pending && state == DAEMON_IDLE &&
+		if (reconfigure_pending != FORCE_RELOAD_NONE &&
+		    state == DAEMON_IDLE &&
 		    (old_state == DAEMON_CONFIGURE ||
 		     old_state == DAEMON_RUNNING)) {
 			/*
@@ -317,7 +322,8 @@ static void __post_config_state(enum daemon_status state)
 			state = DAEMON_CONFIGURE;
 		}
 		if (state == DAEMON_CONFIGURE) {
-			reconfigure_pending = false;
+			reload_type = (reconfigure_pending == FORCE_RELOAD_YES) ? FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
+			reconfigure_pending = FORCE_RELOAD_NONE;
 			__delayed_reconfig = false;
 		}
 		running_state = state;
@@ -334,20 +340,26 @@ void post_config_state(enum daemon_status state)
 	pthread_cleanup_pop(1);
 }
 
-void schedule_reconfigure(void)
+void schedule_reconfigure(enum force_reload_types requested_type)
 {
 	pthread_mutex_lock(&config_lock);
 	pthread_cleanup_push(config_cleanup, NULL);
+	enum force_reload_types type;
+
+	type = (reconfigure_pending == FORCE_RELOAD_YES ||
+		requested_type == FORCE_RELOAD_YES) ?
+	       FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
 	switch (running_state)
 	{
 	case DAEMON_SHUTDOWN:
 		break;
 	case DAEMON_IDLE:
+		reconfigure_pending = type;
 		__post_config_state(DAEMON_CONFIGURE);
 		break;
 	case DAEMON_CONFIGURE:
 	case DAEMON_RUNNING:
-		reconfigure_pending = true;
+		reconfigure_pending = type;
 		break;
 	default:
 		break;
@@ -812,7 +824,7 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 			if (get_delayed_reconfig() &&
 			    !need_to_delay_reconfig(vecs)) {
 				condlog(2, "reconfigure (delayed)");
-				schedule_reconfigure();
+				schedule_reconfigure(FORCE_RELOAD_WEAK);
 				return 0;
 			}
 		}
@@ -1830,7 +1842,7 @@ missing_uev_wait_tick(struct vectors *vecs)
 	if (timed_out && get_delayed_reconfig() &&
 	    !need_to_delay_reconfig(vecs)) {
 		condlog(2, "reconfigure (delayed)");
-		schedule_reconfigure();
+		schedule_reconfigure(FORCE_RELOAD_WEAK);
 	}
 }
 
@@ -2595,7 +2607,6 @@ configure (struct vectors * vecs)
 	vector mpvec;
 	int i, ret;
 	struct config *conf;
-	static int force_reload = FORCE_RELOAD_WEAK;
 
 	if (!vecs->pathvec && !(vecs->pathvec = vector_alloc())) {
 		condlog(0, "couldn't allocate path vec in configure");
@@ -2643,15 +2654,7 @@ configure (struct vectors * vecs)
 	if (should_exit())
 		goto fail;
 
-	/*
-	 * create new set of maps & push changed ones into dm
-	 * In the first call, use FORCE_RELOAD_WEAK to avoid making
-	 * superfluous ACT_RELOAD ioctls. Later calls are done
-	 * with FORCE_RELOAD_YES.
-	 */
-	ret = coalesce_paths(vecs, mpvec, NULL, force_reload, CMD_NONE);
-	if (force_reload == FORCE_RELOAD_WEAK)
-		force_reload = FORCE_RELOAD_YES;
+	ret = coalesce_paths(vecs, mpvec, NULL, reload_type, CMD_NONE);
 	if (ret != CP_OK) {
 		condlog(0, "configure failed while coalescing paths");
 		goto fail;
@@ -2815,7 +2818,7 @@ handle_signals(bool nonfatal)
 		return;
 	if (reconfig_sig) {
 		condlog(2, "reconfigure (signal)");
-		schedule_reconfigure();
+		schedule_reconfigure(FORCE_RELOAD_YES);
 	}
 	if (log_reset_sig) {
 		condlog(2, "reset log (signal)");
@@ -3276,7 +3279,7 @@ child (__attribute__((unused)) void *param)
 			if (!need_to_delay_reconfig(vecs))
 				reconfigure(vecs);
 			else
-				set_delayed_reconfig(true);
+				enable_delayed_reconfig();
 			lock_cleanup_pop(vecs->lock);
 			post_config_state(DAEMON_IDLE);
 		}
diff --git a/multipathd/main.h b/multipathd/main.h
index 7017380..c91f9f9 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -37,7 +37,7 @@ void exit_daemon(void);
 const char * daemon_status(void);
 enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 					    unsigned long ms);
-void schedule_reconfigure(void);
+void schedule_reconfigure(enum force_reload_types requested_type);
 int need_to_delay_reconfig (struct vectors *);
 int ev_add_path (struct path *, struct vectors *, int);
 int ev_remove_path (struct path *, struct vectors *, int);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 39/48] multipathd: add "reconfigure all" command.

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

With this commit, multipathd no longer defaults to full reconfigures for
the "reconfigure" command and the HUP signal. The default is a weak
reconfigure. A new command, "reconfigure all", has been added to do
a full reconfigure.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipath/main.c          |  2 +-
 multipathd/cli.c          |  1 +
 multipathd/cli.h          |  2 ++
 multipathd/cli_handlers.c | 10 ++++++++++
 multipathd/main.c         |  2 +-
 multipathd/multipathd.8   | 10 ++++++++--
 6 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/multipath/main.c b/multipath/main.c
index b2d300e..47e799b 100644
--- a/multipath/main.c
+++ b/multipath/main.c
@@ -760,7 +760,7 @@ int delegate_to_multipathd(enum mpath_cmds cmd,
 		return NOT_DELEGATED;
 
 	if (cmd == CMD_CREATE && conf->force_reload == FORCE_RELOAD_YES) {
-		p += snprintf(p, n, "reconfigure");
+		p += snprintf(p, n, "reconfigure all");
 	}
 	else if (cmd == CMD_FLUSH_ONE && dev && dev_type == DEV_DEVMAP) {
 		p += snprintf(p, n, "del map %s", dev);
diff --git a/multipathd/cli.c b/multipathd/cli.c
index 1c6351e..b42db81 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -210,6 +210,7 @@ load_keys (void)
 	r += add_key(keys, "local", LOCAL, 0);
 	r += add_key(keys, "setmarginal", SETMARGINAL, 0);
 	r += add_key(keys, "unsetmarginal", UNSETMARGINAL, 0);
+	r += add_key(keys, "all", ALL, 0);
 
 
 	if (r) {
diff --git a/multipathd/cli.h b/multipathd/cli.h
index b05746f..21685fc 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -47,6 +47,7 @@ enum {
 	__LOCAL,			/* 40 */
 	__SETMARGINAL,
 	__UNSETMARGINAL,
+	__ALL,
 };
 
 #define LIST		(1ULL << __LIST)
@@ -93,6 +94,7 @@ enum {
 #define LOCAL		(1ULL << __LOCAL)
 #define SETMARGINAL	(1ULL << __SETMARGINAL)
 #define UNSETMARGINAL	(1ULL << __UNSETMARGINAL)
+#define ALL		(1ULL << __ALL)
 
 #define INITIAL_REPLY_LEN	1200
 
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index d165675..f89f4e7 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -1012,6 +1012,15 @@ cli_reconfigure(void * v, struct strbuf *reply, void * data)
 {
 	condlog(2, "reconfigure (operator)");
 
+	schedule_reconfigure(FORCE_RELOAD_WEAK);
+	return 0;
+}
+
+int
+cli_reconfigure_all(void * v, struct strbuf *reply, void * data)
+{
+	condlog(2, "reconfigure all (operator)");
+
 	schedule_reconfigure(FORCE_RELOAD_YES);
 	return 0;
 }
@@ -1497,6 +1506,7 @@ void init_handler_callbacks(void)
 	set_handler_callback(DEL+MAPS, cli_del_maps);
 	set_handler_callback(SWITCH+MAP+GROUP, cli_switch_group);
 	set_unlocked_handler_callback(RECONFIGURE, cli_reconfigure);
+	set_unlocked_handler_callback(RECONFIGURE+ALL, cli_reconfigure_all);
 	set_handler_callback(SUSPEND+MAP, cli_suspend);
 	set_handler_callback(RESUME+MAP, cli_resume);
 	set_handler_callback(RESIZE+MAP, cli_resize);
diff --git a/multipathd/main.c b/multipathd/main.c
index 1efbaab..f8a422a 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -2818,7 +2818,7 @@ handle_signals(bool nonfatal)
 		return;
 	if (reconfig_sig) {
 		condlog(2, "reconfigure (signal)");
-		schedule_reconfigure(FORCE_RELOAD_YES);
+		schedule_reconfigure(FORCE_RELOAD_WEAK);
 	}
 	if (log_reset_sig) {
 		condlog(2, "reset log (signal)");
diff --git a/multipathd/multipathd.8 b/multipathd/multipathd.8
index 048a838..e60d7f7 100644
--- a/multipathd/multipathd.8
+++ b/multipathd/multipathd.8
@@ -195,10 +195,16 @@ group index, starting with 1.
 .
 .TP
 .B reconfigure
-Reconfigures the multipaths. This should be triggered automatically after anyi
-hotplug event.
+Rereads the configuration, and reloads all changed multipath devices. This
+also happens at startup, when the service is reload, or when a SIGHUP is
+received.
 .
 .TP
+.B reconfigure all
+Rereads the configuration, and reloads all multipath devices regardless of
+whether or not they have changed. This also happens when \fImultipath -r\fR is
+run.
+.TP
 .B suspend map|multipath $map
 Sets map $map into suspend state.
 .
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 40/48] multipathd: remove missing paths on startup

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

If a path device was removed from the system while multipathd was not
running, multipathd would not remove the path from the multipath table
on start-up, or on a weak reconfigure. update_pathvec_from_dm() would
return that a reload was necessary, but that information wasn't
propigated back to where it could be used to reload the device.

Multipath devices now remember if they need to be reloaded, and if so,
force_reload is set in select_action().  This means that even when
configure is called with FORCE_RELOAD_WEAK, these devices will still be
reloaded.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/configure.c   | 2 ++
 libmultipath/devmapper.c   | 2 ++
 libmultipath/structs.h     | 1 +
 libmultipath/structs_vec.c | 1 +
 4 files changed, 6 insertions(+)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index eb8ec1b..f1a890a 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -715,6 +715,8 @@ void select_action (struct multipath *mpp, const struct _vector *curmp,
 
 	cmpp = find_mp_by_wwid(curmp, mpp->wwid);
 	cmpp_by_name = find_mp_by_alias(curmp, mpp->alias);
+	if (mpp->need_reload || (cmpp && cmpp->need_reload))
+		force_reload = 1;
 
 	if (!cmpp_by_name) {
 		if (cmpp) {
diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index c05dc20..3e1a726 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -522,6 +522,8 @@ freeout:
 addout:
 	dm_task_destroy (dmt);
 
+	if (r)
+		mpp->need_reload = false;
 	return r;
 }
 
diff --git a/libmultipath/structs.h b/libmultipath/structs.h
index 399540e..d0b266b 100644
--- a/libmultipath/structs.h
+++ b/libmultipath/structs.h
@@ -355,6 +355,7 @@ struct multipath {
 	int retain_hwhandler;
 	int deferred_remove;
 	bool in_recovery;
+	bool need_reload;
 	int san_path_err_threshold;
 	int san_path_err_forget_rate;
 	int san_path_err_recovery_time;
diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index 85d97ac..e52db0c 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -237,6 +237,7 @@ bool update_pathvec_from_dm(vector pathvec, struct multipath *mpp,
 		free_pathgroup(pgp, KEEP_PATHS);
 		must_reload = true;
 	}
+	mpp->need_reload = mpp->need_reload || must_reload;
 	return must_reload;
 }
 
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 41/48] libmultipath: skip unneeded steps to get path name

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

The path already must have a udev device at this point, so it just
needs to copy the sysname from it.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/structs_vec.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index e52db0c..4d56107 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -1,6 +1,7 @@
 #include <stdio.h>
 #include <string.h>
 #include <unistd.h>
+#include <libudev.h>
 
 #include "util.h"
 #include "checkers.h"
@@ -174,8 +175,8 @@ bool update_pathvec_from_dm(vector pathvec, struct multipath *mpp,
 				} else {
 					int rc;
 
-					devt2devname(pp->dev, sizeof(pp->dev),
-						     pp->dev_t);
+					strlcpy(pp->dev, udev_device_get_sysname(pp->udev),
+						sizeof(pp->dev));
 					conf = get_multipath_config();
 					pthread_cleanup_push(put_multipath_config,
 							     conf);
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 42/48] libmultipath: don't use fallback wwid in update_pathvec_from_dm

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

When new paths are added in update_pathvec_from_dm(). If they can't get
their regular wwid, they shouldn't try the getting the fallback wwid,
and should just copy the wwid of the multipath device.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/discovery.c   | 7 ++++---
 libmultipath/discovery.h   | 2 ++
 libmultipath/structs_vec.c | 3 +--
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index f25fe9e..5edf959 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -2363,15 +2363,16 @@ int pathinfo(struct path *pp, struct config *conf, int mask)
 	}
 
 	if ((mask & DI_WWID) && !strlen(pp->wwid)) {
-		get_uid(pp, path_state, pp->udev,
-			(pp->retriggers >= conf->retrigger_tries));
+		int allow_fallback = ((mask & DI_NOFALLBACK) == 0 &&
+				      pp->retriggers >= conf->retrigger_tries);
+		get_uid(pp, path_state, pp->udev, allow_fallback);
 		if (!strlen(pp->wwid)) {
 			if (pp->bus == SYSFS_BUS_UNDEF)
 				return PATHINFO_SKIPPED;
 			if (pp->initialized != INIT_FAILED) {
 				pp->initialized = INIT_MISSING_UDEV;
 				pp->tick = conf->retrigger_delay;
-			} else if (pp->retriggers >= conf->retrigger_tries &&
+			} else if (allow_fallback &&
 				   (pp->state == PATH_UP || pp->state == PATH_GHOST)) {
 				/*
 				 * We have failed to read udev info for this path
diff --git a/libmultipath/discovery.h b/libmultipath/discovery.h
index a5446b4..095657b 100644
--- a/libmultipath/discovery.h
+++ b/libmultipath/discovery.h
@@ -70,6 +70,7 @@ enum discovery_mode {
 	__DI_WWID,
 	__DI_BLACKLIST,
 	__DI_NOIO,
+	__DI_NOFALLBACK,
 };
 
 #define DI_SYSFS	(1 << __DI_SYSFS)
@@ -79,6 +80,7 @@ enum discovery_mode {
 #define DI_WWID		(1 << __DI_WWID)
 #define DI_BLACKLIST	(1 << __DI_BLACKLIST)
 #define DI_NOIO		(1 << __DI_NOIO) /* Avoid IO on the device */
+#define DI_NOFALLBACK	(1 << __DI_NOFALLBACK) /* do not allow wwid fallback */
 
 #define DI_ALL		(DI_SYSFS  | DI_SERIAL | DI_CHECKER | DI_PRIO | \
 			 DI_WWID)
diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index 4d56107..d363e7f 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -182,8 +182,7 @@ bool update_pathvec_from_dm(vector pathvec, struct multipath *mpp,
 							     conf);
 					pp->checkint = conf->checkint;
 					rc = pathinfo(pp, conf,
-						      DI_SYSFS|DI_WWID|DI_BLACKLIST|
-						      pathinfo_flags);
+						      DI_SYSFS|DI_WWID|DI_BLACKLIST|DI_NOFALLBACK|pathinfo_flags);
 					pthread_cleanup_pop(1);
 					if (rc != PATHINFO_OK) {
 						condlog(1, "%s: error %d in pathinfo, discarding path",
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 43/48] libmultipath: always set INIT_REMOVED in set_path_removed

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

Avoiding this corner case simplifies a future patch

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/structs_vec.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index d363e7f..fb26437 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -326,10 +326,8 @@ void set_path_removed(struct path *pp)
 	 * Keep link to mpp. It will be removed when the path
 	 * is successfully removed from the map.
 	 */
-	if (!mpp) {
+	if (!mpp)
 		condlog(0, "%s: internal error: mpp == NULL", pp->dev);
-		return;
-	}
 	pp->mpp = mpp;
 	pp->initialized = INIT_REMOVED;
 }
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 44/48] multipathd: fully initialize paths added by update_pathvec_from_dm

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

When paths are added by update_pathvec_from_dm(), udev may not have
initialized them. This means that it's possible that they are supposed
to be blacklisted by udev properties, but weren't.  Also, in order to
avoid doing potentially stalling IO, update_pathvec_from_dm() doesn't
get all the path information in pathinfo(). These paths end up in the
unexpected state of INIT_MISSING_UDEV or INIT_NEW, but with their mpp
and wwid set.

If udev has already initialized the path, but multipathed wasn't
monitoring it, the blacklist checks and wwid determination in
update_pathvec_from_dm() will work correctly, so paths added by it are
safe, but not completely initialized. The most likely reason why this
would happen is if the path was manually removed from multipathd
monitoring with "multipathd del path". The most common time when
uninitialized paths would in be part of multipath devices is during
boot, after the pivot root, but before the udev coldplug happens. These
paths are not necessarily safe. It's possible that /etc/multipath.conf
in the initramfs and regular filesystem differ, and they should now be
either blacklisted by udev_property, or have a different wwid. However
an "add" event should appear for them shortly.

Multipath now has a new state to deal with these devices, INIT_PARTIAL.
Devices in this state are treated mostly like INIT_OK devices, but
when "multipathd add path" is called or an add/change uevent happens
on these devices, multipathd will finish initializing them, and remove
them if necessary.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/structs.h     |  6 +++++
 libmultipath/structs_vec.c |  5 ++--
 multipathd/cli_handlers.c  | 35 ++++++++++++++++++++++++--
 multipathd/main.c          | 51 +++++++++++++++++++++++++++++++++++---
 multipathd/main.h          |  1 +
 5 files changed, 90 insertions(+), 8 deletions(-)

diff --git a/libmultipath/structs.h b/libmultipath/structs.h
index d0b266b..69409fd 100644
--- a/libmultipath/structs.h
+++ b/libmultipath/structs.h
@@ -200,6 +200,12 @@ enum initialized_states {
 	 * mapped by some multipath map because of map reload failure.
 	 */
 	INIT_REMOVED,
+	/*
+	 * INIT_PARTIAL: paths added by update_pathvec_from_dm() will not
+	 * be fully initialized. This will be handled when an add or
+	 * change uevent is received.
+	 */
+	INIT_PARTIAL,
 };
 
 enum prkey_sources {
diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index fb26437..1de9175 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -194,6 +194,7 @@ bool update_pathvec_from_dm(vector pathvec, struct multipath *mpp,
 					}
 					condlog(2, "%s: adding new path %s",
 						mpp->alias, pp->dev);
+					pp->initialized = INIT_PARTIAL;
 					store_path(pathvec, pp);
 					pp->tick = 1;
 				}
@@ -392,12 +393,12 @@ extract_hwe_from_path(struct multipath * mpp)
 	condlog(4, "%s: searching paths for valid hwe", mpp->alias);
 	/* doing this in two passes seems like paranoia to me */
 	vector_foreach_slot(mpp->paths, pp, i) {
-		if (pp->state == PATH_UP &&
+		if (pp->state == PATH_UP && pp->initialized != INIT_PARTIAL &&
 		    pp->initialized != INIT_REMOVED && pp->hwe)
 			goto done;
 	}
 	vector_foreach_slot(mpp->paths, pp, i) {
-		if (pp->state != PATH_UP &&
+		if ((pp->state != PATH_UP || pp->initialized == INIT_PARTIAL) &&
 		    pp->initialized != INIT_REMOVED && pp->hwe)
 			goto done;
 	}
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index f89f4e7..8bd407c 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -493,6 +493,33 @@ cli_reset_map_stats (void *v, struct strbuf *reply, void *data)
 	return 0;
 }
 
+static int
+add_partial_path(struct path *pp, struct vectors *vecs)
+{
+	char wwid[WWID_SIZE];
+	struct udev_device *udd;
+
+	udd = get_udev_device(pp->dev_t, DEV_DEVT);
+	if (!udd)
+		return 0;
+	strcpy(wwid, pp->wwid);
+	if (get_uid(pp, pp->state, udd, 0) != 0) {
+		strcpy(pp->wwid, wwid);
+		udev_device_unref(udd);
+		return 0;
+	}
+	if (strlen(wwid) && strncmp(wwid, pp->wwid, WWID_SIZE) != 0) {
+		condlog(0, "%s: path wwid changed from '%s' to '%s'. removing",
+			pp->dev, wwid, pp->wwid);
+		ev_remove_path(pp, vecs, 1);
+		udev_device_unref(udd);
+		return -1;
+	}
+	udev_device_unref(pp->udev);
+	pp->udev = udd;
+	return finish_path_init(pp, vecs);
+}
+
 static int
 cli_add_path (void *v, struct strbuf *reply, void *data)
 {
@@ -518,8 +545,12 @@ cli_add_path (void *v, struct strbuf *reply, void *data)
 	if (pp && pp->initialized != INIT_REMOVED) {
 		condlog(2, "%s: path already in pathvec", param);
 
-		if (pp->recheck_wwid == RECHECK_WWID_ON &&
-		    check_path_wwid_change(pp)) {
+		if (pp->initialized == INIT_PARTIAL) {
+			if (add_partial_path(pp, vecs) < 0)
+				return 1;
+		}
+		else if (pp->recheck_wwid == RECHECK_WWID_ON &&
+			 check_path_wwid_change(pp)) {
 			condlog(0, "%s: wwid changed. Removing device",
 				pp->dev);
 			handle_path_wwid_change(pp, vecs);
diff --git a/multipathd/main.c b/multipathd/main.c
index f8a422a..828d229 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -977,12 +977,19 @@ check_path_wwid_change(struct path *pp)
 	return false;
 }
 
+/*
+ * uev_add_path can call uev_update_path, and uev_update_path can call
+ * uev_add_path
+ */
+static int uev_update_path (struct uevent *uev, struct vectors * vecs);
+
 static int
 uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
 {
 	struct path *pp;
 	int ret = 0, i;
 	struct config *conf;
+	bool partial_init = false;
 
 	condlog(3, "%s: add path (uevent)", uev->kernel);
 	if (strstr(uev->kernel, "..") != NULL) {
@@ -1001,7 +1008,10 @@ uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
 		int r;
 		struct multipath *prev_mpp = NULL;
 
-		if (pp->initialized == INIT_REMOVED) {
+		if (pp->initialized == INIT_PARTIAL) {
+			partial_init = true;
+			goto out;
+		} else if (pp->initialized == INIT_REMOVED) {
 			condlog(3, "%s: re-adding removed path", pp->dev);
 			pp->initialized = INIT_NEW;
 			prev_mpp = pp->mpp;
@@ -1111,6 +1121,8 @@ uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
 	}
 out:
 	lock_cleanup_pop(vecs->lock);
+	if (partial_init)
+		return uev_update_path(uev, vecs);
 	return ret;
 }
 
@@ -1406,6 +1418,28 @@ fail:
 	return REMOVE_PATH_MAP_ERROR;
 }
 
+int
+finish_path_init(struct path *pp, struct vectors * vecs)
+{
+	int r;
+	struct config *conf;
+
+	if (pp->udev && pp->uid_attribute && *pp->uid_attribute &&
+	    !udev_device_get_is_initialized(pp->udev))
+		return 0;
+	conf = get_multipath_config();
+	pthread_cleanup_push(put_multipath_config, conf);
+	r = pathinfo(pp, conf, DI_ALL|DI_BLACKLIST);
+	pthread_cleanup_pop(1);
+
+	if (r == PATHINFO_OK)
+		return 0;
+
+	condlog(0, "%s: error fully initializing path, removing", pp->dev);
+	ev_remove_path(pp, vecs, 1);
+	return -1;
+}
+
 static int
 uev_update_path (struct uevent *uev, struct vectors * vecs)
 {
@@ -1444,7 +1478,7 @@ uev_update_path (struct uevent *uev, struct vectors * vecs)
 		}
 		/* Don't deal with other types of failed initialization
 		 * now. check_path will handle it */
-		if (!strlen(pp->wwid))
+		if (!strlen(pp->wwid) && pp->initialized != INIT_PARTIAL)
 			goto out;
 
 		strcpy(wwid, pp->wwid);
@@ -1452,12 +1486,20 @@ uev_update_path (struct uevent *uev, struct vectors * vecs)
 
 		if (rc != 0)
 			strcpy(pp->wwid, wwid);
-		else if (strncmp(wwid, pp->wwid, WWID_SIZE) != 0) {
+		else if (strlen(wwid) &&
+			 strncmp(wwid, pp->wwid, WWID_SIZE) != 0) {
 			condlog(0, "%s: path wwid changed from '%s' to '%s'",
 				uev->kernel, wwid, pp->wwid);
 			ev_remove_path(pp, vecs, 1);
 			needs_reinit = 1;
 			goto out;
+		} else if (pp->initialized == INIT_PARTIAL) {
+			udev_device_unref(pp->udev);
+			pp->udev = udev_device_ref(uev->udev);
+			if (finish_path_init(pp, vecs) < 0) {
+				retval = 1;
+				goto out;
+			}
 		} else {
 			udev_device_unref(pp->udev);
 			pp->udev = udev_device_ref(uev->udev);
@@ -1508,6 +1550,7 @@ out:
 
 		condlog(0, "%s: spurious uevent, path not found", uev->kernel);
 	}
+	/* pp->initalized must not be INIT_PARTIAL if needs_reinit is set */
 	if (needs_reinit)
 		retval = uev_add_path(uev, vecs, 1);
 	return retval;
@@ -2117,7 +2160,7 @@ check_path (struct vectors * vecs, struct path * pp, unsigned int ticks)
 	int marginal_pathgroups, marginal_changed = 0;
 	int ret;
 
-	if (((pp->initialized == INIT_OK ||
+	if (((pp->initialized == INIT_OK || pp->initialized == INIT_PARTIAL ||
 	      pp->initialized == INIT_REQUESTED_UDEV) && !pp->mpp) ||
 	    pp->initialized == INIT_REMOVED)
 		return 0;
diff --git a/multipathd/main.h b/multipathd/main.h
index c91f9f9..8356b25 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -65,4 +65,5 @@ int reload_and_sync_map(struct multipath *mpp, struct vectors *vecs,
 
 void handle_path_wwid_change(struct path *pp, struct vectors *vecs);
 bool check_path_wwid_change(struct path *pp);
+int finish_path_init(struct path *pp, struct vectors * vecs);
 #endif /* MAIN_H */
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 45/48] multipathd: retrigger uevent for partial paths

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

If a partial path appears and is not fully initialized within
180 seconds, trigger a uevent. If the udev device is not initialized
trigger an add event. Otherwise, trigger a change event.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/libmultipath.version |  2 +-
 libmultipath/structs.h            |  1 +
 libmultipath/structs_vec.c        |  1 +
 multipathd/main.c                 | 20 ++++++++++++++++++++
 4 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index 4f6fa1d..5a3ba4e 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -31,7 +31,7 @@
  *   The new version inherits the previous ones.
  */
 
-LIBMULTIPATH_11.0.0 {
+LIBMULTIPATH_12.0.0 {
 global:
 	/* symbols referenced by multipath and multipathd */
 	add_foreign;
diff --git a/libmultipath/structs.h b/libmultipath/structs.h
index 69409fd..c21d1ed 100644
--- a/libmultipath/structs.h
+++ b/libmultipath/structs.h
@@ -318,6 +318,7 @@ struct path {
 	int fd;
 	int initialized;
 	int retriggers;
+	int partial_retrigger_delay;
 	unsigned int path_failures;
 	time_t dis_reinstate_time;
 	int disable_reinstate;
diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index 1de9175..9b6407b 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -195,6 +195,7 @@ bool update_pathvec_from_dm(vector pathvec, struct multipath *mpp,
 					condlog(2, "%s: adding new path %s",
 						mpp->alias, pp->dev);
 					pp->initialized = INIT_PARTIAL;
+					pp->partial_retrigger_delay = 180;
 					store_path(pathvec, pp);
 					pp->tick = 1;
 				}
diff --git a/multipathd/main.c b/multipathd/main.c
index 828d229..bffafe9 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1949,6 +1949,25 @@ retry_count_tick(vector mpvec)
 	}
 }
 
+static void
+partial_retrigger_tick(vector pathvec)
+{
+	struct path *pp;
+	unsigned int i;
+
+	vector_foreach_slot (pathvec, pp, i) {
+		if (pp->initialized == INIT_PARTIAL && pp->udev &&
+		    pp->partial_retrigger_delay > 0 &&
+		    --pp->partial_retrigger_delay == 0) {
+			const char *msg = udev_device_get_is_initialized(pp->udev) ?
+					  "change" : "add";
+
+			sysfs_attr_set_value(pp->udev, "uevent", msg,
+					     strlen(msg));
+		}
+	}
+}
+
 int update_prio(struct path *pp, int refresh_all)
 {
 	int oldpriority;
@@ -2567,6 +2586,7 @@ checkerloop (void *ap)
 		retry_count_tick(vecs->mpvec);
 		missing_uev_wait_tick(vecs);
 		ghost_delay_tick(vecs);
+		partial_retrigger_tick(vecs->pathvec);
 		lock_cleanup_pop(vecs->lock);
 
 		if (count)
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 46/48] multipathd: remove INIT_PARTIAL paths that aren't in a multipath device

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

The only reason multipath is monitoring an INIT_PARTIAL path is because
it was discovered in a multipath device table. If it stops being part
of a multipath device before it gets fully initialized, drop it.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/structs_vec.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index 9b6407b..df5709a 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -308,9 +308,12 @@ void orphan_paths(vector pathvec, struct multipath *mpp, const char *reason)
 
 	vector_foreach_slot (pathvec, pp, i) {
 		if (pp->mpp == mpp) {
-			if (pp->initialized == INIT_REMOVED) {
-				condlog(3, "%s: freeing path in removed state",
-					pp->dev);
+			if (pp->initialized == INIT_REMOVED ||
+			    pp->initialized == INIT_PARTIAL) {
+				condlog(3, "%s: freeing path in %s state",
+					pp->dev,
+					pp->initialized == INIT_REMOVED ?
+					"removed" : "partial");
 				vector_del_slot(pathvec, i--);
 				free_path(pp);
 			} else
@@ -469,11 +472,14 @@ static void check_removed_paths(const struct multipath *mpp, vector pathvec)
 	int i;
 
 	vector_foreach_slot(pathvec, pp, i) {
-		if (pp->initialized != INIT_REMOVED || pp->mpp != mpp)
-			continue;
-		if (!find_devt_in_pathgroups(mpp, pp->dev_t)) {
-			condlog(2, "%s: %s: freeing path in removed state",
-				__func__, pp->dev);
+		if (pp->mpp == mpp &&
+		    (pp->initialized == INIT_REMOVED ||
+		     pp->initialized == INIT_PARTIAL) &&
+		    !find_devt_in_pathgroups(mpp, pp->dev_t)) {
+			condlog(2, "%s: %s: freeing path in %s state",
+				__func__, pp->dev,
+				pp->initialized == INIT_REMOVED ?
+				"removed" : "partial");
 			vector_del_slot(pathvec, i--);
 			free_path(pp);
 		}
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 47/48] multipathd: Remove dependency on systemd-udev-settle.service

[mwilck]

From: Benjamin Marzinski <bmarzins@redhat.com>

multipathd can now handle starting up with incompletely initialized
paths, so it no longer needs to wait for the device Coldplug to
complete. However multipathd may need to write to /etc (for the wwids
and bindings files), so in needs to wait for the root filesystem to
be remounted read/write before starting.

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 multipathd/multipathd.service | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/multipathd/multipathd.service b/multipathd/multipathd.service
index 0b2ac81..87cb534 100644
--- a/multipathd/multipathd.service
+++ b/multipathd/multipathd.service
@@ -1,9 +1,8 @@
 [Unit]
 Description=Device-Mapper Multipath Device Controller
-Wants=systemd-udev-trigger.service systemd-udev-settle.service
 Before=iscsi.service iscsid.service lvm2-activation-early.service
 Before=local-fs-pre.target blk-availability.service shutdown.target
-After=multipathd.socket systemd-udev-trigger.service systemd-udev-settle.service
+After=multipathd.socket systemd-remount-fs.service
 DefaultDependencies=no
 Conflicts=shutdown.target
 ConditionKernelCommandLine=!nompath
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v2 48/48] libmultipath: add path wildcard "%I" for init state

[mwilck]

From: Martin Wilck <mwilck@suse.com>

Enable printing pp->initialized with 'multipathd show paths format "%I"'.
This is supposed to go on top of Ben's "multipathd: remove udev settle
dependency" series, to simplify checking multipathd's state.

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 libmultipath/print.c   | 21 +++++++++++++++++++++
 libmultipath/structs.h |  1 +
 2 files changed, 22 insertions(+)

diff --git a/libmultipath/print.c b/libmultipath/print.c
index d2ef010..e61349f 100644
--- a/libmultipath/print.c
+++ b/libmultipath/print.c
@@ -504,6 +504,26 @@ snprint_dm_path_state (struct strbuf *buff, const struct path * pp)
 	}
 }
 
+static int snprint_initialized(struct strbuf *buff, const struct path * pp)
+{
+	static const char *init_state_name[] = {
+		[INIT_NEW] = "new",
+		[INIT_FAILED] = "failed",
+		[INIT_MISSING_UDEV] = "udev-missing",
+		[INIT_REQUESTED_UDEV] = "udev-requested",
+		[INIT_OK] = "ok",
+		[INIT_REMOVED] = "removed",
+		[INIT_PARTIAL] = "partial",
+	};
+	const char *str;
+
+	if (pp->initialized < INIT_NEW || pp->initialized >= __INIT_LAST)
+		str = "undef";
+	else
+		str = init_state_name[pp->initialized];
+	return append_strbuf_str(buff, str);
+}
+
 static int
 snprint_vpr (struct strbuf *buff, const struct path * pp)
 {
@@ -804,6 +824,7 @@ struct path_data pd[] = {
 	{'g', "vpd page data", 0, snprint_path_vpd_data},
 	{'0', "failures",      0, snprint_path_failures},
 	{'P', "protocol",      0, snprint_path_protocol},
+	{'I', "init_st",       0, snprint_initialized},
 	{0, NULL, 0 , NULL}
 };
 
diff --git a/libmultipath/structs.h b/libmultipath/structs.h
index c21d1ed..c0f8929 100644
--- a/libmultipath/structs.h
+++ b/libmultipath/structs.h
@@ -206,6 +206,7 @@ enum initialized_states {
 	 * change uevent is received.
 	 */
 	INIT_PARTIAL,
+	__INIT_LAST,
 };
 
 enum prkey_sources {
-- 
2.33.1


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:57:55PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Have struct mutex_lock take an optional wakeup function.
> unlock() is renamed to __unlock() in order to prevent it from
> being called by mistake.
> 
> This changes offsets in "struct vectors", requiring a major
> libmultipath version bump. While the strucure is already changed,
> in order to avoid this in the future, move the lock to the end
> of "struct vectors".
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>

In libmultipath.version, I would have said that set_wakeup_fn was added
in 10.0.0, instead of 9.2.0, which is a version that never actually
existed, but I don't think that's going to cause any problems so,

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> ---
>  libmultipath/libmultipath.version | 13 +++++++------
>  libmultipath/lock.c               | 12 +++++++++++-
>  libmultipath/lock.h               |  6 +++++-
>  libmultipath/structs_vec.h        |  2 +-
>  4 files changed, 24 insertions(+), 9 deletions(-)
> 
> diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
> index c98cf7f..98ec502 100644
> --- a/libmultipath/libmultipath.version
> +++ b/libmultipath/libmultipath.version
> @@ -31,7 +31,7 @@
>   *   The new version inherits the previous ones.
>   */
>  
> -LIBMULTIPATH_9.0.0 {
> +LIBMULTIPATH_10.0.0 {
>  global:
>  	/* symbols referenced by multipath and multipathd */
>  	add_foreign;
> @@ -284,11 +284,12 @@ global:
>  	/* added in 8.2.0 */
>  	check_daemon;
>  
> +	/* added in 9.1.0 */
> +	timespeccmp;
> +
> +	/* added in 9.2.0 */
> +	set_wakeup_fn;
> +
>  local:
>  	*;
>  };
> -
> -LIBMULTIPATH_9.1.0 {
> -global:
> -	timespeccmp;
> -} LIBMULTIPATH_9.0.0;
> diff --git a/libmultipath/lock.c b/libmultipath/lock.c
> index 72c70e3..93b48db 100644
> --- a/libmultipath/lock.c
> +++ b/libmultipath/lock.c
> @@ -3,6 +3,16 @@
>  void cleanup_lock (void * data)
>  {
>  	struct mutex_lock *lock = data;
> +	wakeup_fn *fn = lock->wakeup;
>  
> -	unlock(lock);
> +	__unlock(lock);
> +	if (fn)
> +		fn();
> +}
> +
> +void set_wakeup_fn(struct mutex_lock *lck, wakeup_fn *fn)
> +{
> +	lock(lck);
> +	lck->wakeup = fn;
> +	__unlock(lck);
>  }
> diff --git a/libmultipath/lock.h b/libmultipath/lock.h
> index d99eedb..d7b779e 100644
> --- a/libmultipath/lock.h
> +++ b/libmultipath/lock.h
> @@ -3,8 +3,11 @@
>  
>  #include <pthread.h>
>  
> +typedef void (wakeup_fn)(void);
> +
>  struct mutex_lock {
>  	pthread_mutex_t mutex;
> +	wakeup_fn *wakeup;
>  };
>  
>  static inline void lock(struct mutex_lock *a)
> @@ -22,7 +25,7 @@ static inline int timedlock(struct mutex_lock *a, struct timespec *tmo)
>  	return pthread_mutex_timedlock(&a->mutex, tmo);
>  }
>  
> -static inline void unlock(struct mutex_lock *a)
> +static inline void __unlock(struct mutex_lock *a)
>  {
>  	pthread_mutex_unlock(&a->mutex);
>  }
> @@ -30,5 +33,6 @@ static inline void unlock(struct mutex_lock *a)
>  #define lock_cleanup_pop(a) pthread_cleanup_pop(1)
>  
>  void cleanup_lock (void * data);
> +void set_wakeup_fn(struct mutex_lock *lock, wakeup_fn *fn);
>  
>  #endif /* _LOCK_H */
> diff --git a/libmultipath/structs_vec.h b/libmultipath/structs_vec.h
> index 29ede45..2a0cbd1 100644
> --- a/libmultipath/structs_vec.h
> +++ b/libmultipath/structs_vec.h
> @@ -6,9 +6,9 @@
>  #include "lock.h"
>  
>  struct vectors {
> -	struct mutex_lock lock; /* defined in lock.h */
>  	vector pathvec;
>  	vector mpvec;
> +	struct mutex_lock lock; /* defined in lock.h */
>  };
>  
>  void __set_no_path_retry(struct multipath *mpp, bool check_features);
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 07/48] multipathd: improve delayed reconfigure

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:57:59PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> When a reconfigure operation is requested, either by the admin
> or by some condition multipathd encounters, the current code
> attempts to set DAEMON_CONFIGURE state and gives up after a second
> if it doesn't succeed. Apart from shutdown, this happens only
> if multipathd is either already reconfiguring, or busy in the
> path checker loop.
> 
> This patch modifies the logic as follows: rather than waiting,
> we set a flag that requests a reconfigure operation asap, i.e.
> when the current operation is finished and the status switched
> to DAEMON_IDLE. In this case, multipathd will not switch to IDLE
> but start another reconfigure cycle.
> 
> This assumes that if a reconfigure is requested while one is already
> running, the admin has made some (additional) changes and wants
> multipathd to pull them in. As we can't be sure that the currently
> running reconfigure has seen the configuration changes, we need
> to start over again.
> 
> A positive side effect is less waiting in clients and multipathd.
> 
> After this change, the only caller of set_config_state() is
> checkerloop(). Waking up every second just to see that DAEMON_RUNNING
> couldn't be set makes no sense. Therefore set_config_state() is
> changed to wait "forever", or until shutdown is requested. Unless
> multipathd completely hangs, the wait will terminate sooner or
> later.
> 
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/cli_handlers.c | 10 +----
>  multipathd/main.c         | 92 +++++++++++++++++++++++++++++----------
>  multipathd/main.h         |  2 +-
>  3 files changed, 70 insertions(+), 34 deletions(-)
> 
> diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
> index 6d3a0ae..44f76ee 100644
> --- a/multipathd/cli_handlers.c
> +++ b/multipathd/cli_handlers.c
> @@ -1076,17 +1076,9 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
>  int
>  cli_reconfigure(void * v, char ** reply, int * len, void * data)
>  {
> -	int rc;
> -
>  	condlog(2, "reconfigure (operator)");
>  
> -	rc = set_config_state(DAEMON_CONFIGURE);
> -	if (rc == ETIMEDOUT) {
> -		condlog(2, "timeout starting reconfiguration");
> -		return 1;
> -	} else if (rc == EINVAL)
> -		/* daemon shutting down */
> -		return 1;
> +	schedule_reconfigure();
>  	return 0;
>  }
>  
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 1a1812e..7dc5f49 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -221,6 +221,10 @@ static void do_sd_notify(enum daemon_status old_state,
>  	} else if (new_state == DAEMON_CONFIGURE && startup_done)
>  		sd_notify(0, "RELOADING=1");
>  }
> +#else
> +static void do_sd_notify(__attribute__((unused)) enum daemon_status old_state,
> +			 __attribute__((unused)) enum daemon_status new_state)
> +{}
>  #endif
>  
>  static void config_cleanup(__attribute__((unused)) void *arg)
> @@ -266,19 +270,38 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
>  	return st;
>  }
>  
> +/* Don't access this variable without holding config_lock */
> +static bool reconfigure_pending;
> +
>  /* must be called with config_lock held */
>  static void __post_config_state(enum daemon_status state)
>  {
>  	if (state != running_state && running_state != DAEMON_SHUTDOWN) {
> -#ifdef USE_SYSTEMD
>  		enum daemon_status old_state = running_state;
> -#endif
>  
> +		/*
> +		 * Handle a pending reconfigure request.
> +		 * DAEMON_IDLE is set from child() after reconfigure(),
> +		 * or from checkerloop() after completing checkers.
> +		 * In either case, child() will see DAEMON_CONFIGURE
> +		 * again and start another reconfigure cycle.
> +		 */
> +		if (reconfigure_pending && state == DAEMON_IDLE &&
> +		    (old_state == DAEMON_CONFIGURE ||
> +		     old_state == DAEMON_RUNNING)) {
> +			/*
> +			 * notify systemd of transient idle state, lest systemd
> +			 * thinks the reload lasts forever.
> +			 */
> +			do_sd_notify(old_state, DAEMON_IDLE);
> +			old_state = DAEMON_IDLE;
> +			state = DAEMON_CONFIGURE;
> +		}
> +		if (reconfigure_pending && state == DAEMON_CONFIGURE)
> +			reconfigure_pending = false;
>  		running_state = state;
>  		pthread_cond_broadcast(&config_cond);
> -#ifdef USE_SYSTEMD
>  		do_sd_notify(old_state, state);
> -#endif
>  	}
>  }
>  
> @@ -290,24 +313,48 @@ void post_config_state(enum daemon_status state)
>  	pthread_cleanup_pop(1);
>  }
>  
> -int set_config_state(enum daemon_status state)
> +void schedule_reconfigure(void)
> +{
> +	pthread_mutex_lock(&config_lock);
> +	pthread_cleanup_push(config_cleanup, NULL);
> +	switch (running_state)
> +	{
> +	case DAEMON_SHUTDOWN:
> +		break;
> +	case DAEMON_IDLE:
> +		__post_config_state(DAEMON_CONFIGURE);
> +		break;
> +	case DAEMON_CONFIGURE:
> +	case DAEMON_RUNNING:
> +		reconfigure_pending = true;
> +		break;
> +	default:
> +		break;
> +	}
> +	pthread_cleanup_pop(1);
> +}
> +
> +static enum daemon_status set_config_state(enum daemon_status state)
>  {
>  	int rc = 0;
> +	enum daemon_status st;
>  
>  	pthread_cleanup_push(config_cleanup, NULL);
>  	pthread_mutex_lock(&config_lock);
> -	if (running_state != state) {
>  
> -		if (running_state == DAEMON_SHUTDOWN)
> -			rc = EINVAL;
> -		else
> -			rc = __wait_for_state_change(
> -				running_state != DAEMON_IDLE, 1000);
> -		if (!rc)
> -			__post_config_state(state);
> +	while (rc == 0 &&
> +	       running_state != state &&
> +	       running_state != DAEMON_SHUTDOWN &&
> +	       running_state != DAEMON_IDLE) {
> +		rc = pthread_cond_wait(&config_cond, &config_lock);
>  	}
> +
> +	if (rc == 0 && running_state == DAEMON_IDLE && state != DAEMON_IDLE)
> +		__post_config_state(state);
> +	st = running_state;
> +
>  	pthread_cleanup_pop(1);
> -	return rc;
> +	return st;
>  }
>  
>  struct config *get_multipath_config(void)
> @@ -745,7 +792,7 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
>  			if (delayed_reconfig &&
>  			    !need_to_delay_reconfig(vecs)) {
>  				condlog(2, "reconfigure (delayed)");
> -				set_config_state(DAEMON_CONFIGURE);
> +				schedule_reconfigure();
>  				return 0;
>  			}
>  		}
> @@ -1856,7 +1903,7 @@ missing_uev_wait_tick(struct vectors *vecs)
>  	if (timed_out && delayed_reconfig &&
>  	    !need_to_delay_reconfig(vecs)) {
>  		condlog(2, "reconfigure (delayed)");
> -		set_config_state(DAEMON_CONFIGURE);
> +		schedule_reconfigure();
>  	}
>  }
>  
> @@ -2495,6 +2542,10 @@ checkerloop (void *ap)
>  		int num_paths = 0, strict_timing, rc = 0;
>  		unsigned int ticks = 0;
>  
> +		if (set_config_state(DAEMON_RUNNING) != DAEMON_RUNNING)
> +			/* daemon shutdown */
> +			break;
> +
>  		get_monotonic_time(&start_time);
>  		if (start_time.tv_sec && last_time.tv_sec) {
>  			timespecsub(&start_time, &last_time, &diff_time);
> @@ -2510,13 +2561,6 @@ checkerloop (void *ap)
>  		if (use_watchdog)
>  			sd_notify(0, "WATCHDOG=1");
>  #endif
> -		rc = set_config_state(DAEMON_RUNNING);
> -		if (rc == ETIMEDOUT) {
> -			condlog(4, "timeout waiting for DAEMON_IDLE");
> -			continue;
> -		} else if (rc == EINVAL)
> -			/* daemon shutdown */
> -			break;
>  
>  		pthread_cleanup_push(cleanup_lock, &vecs->lock);
>  		lock(&vecs->lock);
> @@ -2844,7 +2888,7 @@ handle_signals(bool nonfatal)
>  		return;
>  	if (reconfig_sig) {
>  		condlog(2, "reconfigure (signal)");
> -		set_config_state(DAEMON_CONFIGURE);
> +		schedule_reconfigure();
>  	}
>  	if (log_reset_sig) {
>  		condlog(2, "reset log (signal)");
> diff --git a/multipathd/main.h b/multipathd/main.h
> index bc1f938..2960a4d 100644
> --- a/multipathd/main.h
> +++ b/multipathd/main.h
> @@ -37,6 +37,7 @@ void exit_daemon(void);
>  const char * daemon_status(void);
>  enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
>  					    unsigned long ms);
> +void schedule_reconfigure(void);
>  int need_to_delay_reconfig (struct vectors *);
>  int reconfigure (struct vectors *);
>  int ev_add_path (struct path *, struct vectors *, int);
> @@ -44,7 +45,6 @@ int ev_remove_path (struct path *, struct vectors *, int);
>  int ev_add_map (char *, const char *, struct vectors *);
>  int ev_remove_map (char *, char *, int, struct vectors *);
>  int flush_map(struct multipath *, struct vectors *, int);
> -int set_config_state(enum daemon_status);
>  void * mpath_alloc_prin_response(int prin_sa);
>  int prin_do_scsi_ioctl(char *, int rq_servact, struct prin_resp * resp,
>  		       int noisy);
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

On Wed, 2021-11-24 at 14:41 -0600, Benjamin Marzinski wrote:
> On Thu, Nov 18, 2021 at 11:57:55PM +0100, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > Have struct mutex_lock take an optional wakeup function.
> > unlock() is renamed to __unlock() in order to prevent it from
> > being called by mistake.
> > 
> > This changes offsets in "struct vectors", requiring a major
> > libmultipath version bump. While the strucure is already changed,
> > in order to avoid this in the future, move the lock to the end
> > of "struct vectors".
> > 
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> 
> In libmultipath.version, I would have said that set_wakeup_fn was
> added
> in 10.0.0, instead of 9.2.0, which is a version that never actually
> existed, but I don't think that's going to cause any problems so,
> 
> Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>

We should consider whether we want bump the version numbers only when
we submit to Christophe, or maybe once per major patch series that
touches the ABI. This is something that I could 

Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 12/48] multipathd: add and set cli_handlers in a single step

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:04PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Modify set_handler_callback() such that a missing slot is created
> if no matching slot is found. This way, we can skip the initialization
> with NULL handlers on startup. Assigning the same handler multiple
> times would be a bug which is tested with assert().
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> ---
>  multipathd/cli.c | 95 ++++++++----------------------------------------
>  multipathd/cli.h |  7 ++--
>  2 files changed, 19 insertions(+), 83 deletions(-)
> 
> diff --git a/multipathd/cli.c b/multipathd/cli.c
> index 5213813..6e5056e 100644
> --- a/multipathd/cli.c
> +++ b/multipathd/cli.c
> @@ -4,6 +4,7 @@
>  #include <sys/time.h>
>  #include <errno.h>
>  #include <pthread.h>
> +#include <assert.h>
>  #include "memory.h"
>  #include "vector.h"
>  #include "structs.h"
> @@ -64,26 +65,26 @@ out:
>  	return 1;
>  }
>  
> -int
> -add_handler (uint64_t fp, cli_handler *fn)
> +static struct handler *add_handler(uint64_t fp, cli_handler *fn, bool locked)
>  {
>  	struct handler * h;
>  
>  	h = alloc_handler();
>  
> -	if (!h)
> -		return 1;
> +	if (h == NULL)
> +		return NULL;
>  
>  	if (!vector_alloc_slot(handlers)) {
>  		FREE(h);
> -		return 1;
> +		return NULL;
>  	}
>  
>  	vector_set_slot(handlers, h);
>  	h->fingerprint = fp;
>  	h->fn = fn;
> +	h->locked = locked;
>  
> -	return 0;
> +	return h;
>  }
>  
>  static struct handler *
> @@ -100,26 +101,17 @@ find_handler (uint64_t fp)
>  }
>  
>  int
> -set_handler_callback (uint64_t fp, cli_handler *fn)
> +__set_handler_callback (uint64_t fp, cli_handler *fn, bool locked)
>  {
> -	struct handler * h = find_handler(fp);
> +	struct handler *h;
>  
> -	if (!h)
> +	assert(find_handler(fp) == NULL);
> +        h = add_handler(fp, fn, locked);
> +        if (!h) {
> +		condlog(0, "%s: failed to set handler for code %"PRIu64,
> +			__func__, fp);
>  		return 1;
> -	h->fn = fn;
> -	h->locked = 1;
> -	return 0;
> -}
> -
> -int
> -set_unlocked_handler_callback (uint64_t fp, cli_handler *fn)
> -{
> -	struct handler * h = find_handler(fp);
> -
> -	if (!h)
> -		return 1;
> -	h->fn = fn;
> -	h->locked = 0;
> +	}
>  	return 0;
>  }
>  
> @@ -513,63 +505,6 @@ cli_init (void) {
>  	if (alloc_handlers())
>  		return 1;
>  
> -	add_handler(LIST+PATHS, NULL);
> -	add_handler(LIST+PATHS+FMT, NULL);
> -	add_handler(LIST+PATHS+RAW+FMT, NULL);
> -	add_handler(LIST+PATH, NULL);
> -	add_handler(LIST+STATUS, NULL);
> -	add_handler(LIST+DAEMON, NULL);
> -	add_handler(LIST+MAPS, NULL);
> -	add_handler(LIST+MAPS+STATUS, NULL);
> -	add_handler(LIST+MAPS+STATS, NULL);
> -	add_handler(LIST+MAPS+FMT, NULL);
> -	add_handler(LIST+MAPS+RAW+FMT, NULL);
> -	add_handler(LIST+MAPS+TOPOLOGY, NULL);
> -	add_handler(LIST+MAPS+JSON, NULL);
> -	add_handler(LIST+TOPOLOGY, NULL);
> -	add_handler(LIST+MAP+TOPOLOGY, NULL);
> -	add_handler(LIST+MAP+JSON, NULL);
> -	add_handler(LIST+MAP+FMT, NULL);
> -	add_handler(LIST+MAP+RAW+FMT, NULL);
> -	add_handler(LIST+CONFIG, NULL);
> -	add_handler(LIST+CONFIG+LOCAL, NULL);
> -	add_handler(LIST+BLACKLIST, NULL);
> -	add_handler(LIST+DEVICES, NULL);
> -	add_handler(LIST+WILDCARDS, NULL);
> -	add_handler(RESET+MAPS+STATS, NULL);
> -	add_handler(RESET+MAP+STATS, NULL);
> -	add_handler(ADD+PATH, NULL);
> -	add_handler(DEL+PATH, NULL);
> -	add_handler(ADD+MAP, NULL);
> -	add_handler(DEL+MAP, NULL);
> -	add_handler(DEL+MAPS, NULL);
> -	add_handler(SWITCH+MAP+GROUP, NULL);
> -	add_handler(RECONFIGURE, NULL);
> -	add_handler(SUSPEND+MAP, NULL);
> -	add_handler(RESUME+MAP, NULL);
> -	add_handler(RESIZE+MAP, NULL);
> -	add_handler(RESET+MAP, NULL);
> -	add_handler(RELOAD+MAP, NULL);
> -	add_handler(DISABLEQ+MAP, NULL);
> -	add_handler(RESTOREQ+MAP, NULL);
> -	add_handler(DISABLEQ+MAPS, NULL);
> -	add_handler(RESTOREQ+MAPS, NULL);
> -	add_handler(REINSTATE+PATH, NULL);
> -	add_handler(FAIL+PATH, NULL);
> -	add_handler(QUIT, NULL);
> -	add_handler(SHUTDOWN, NULL);
> -	add_handler(GETPRSTATUS+MAP, NULL);
> -	add_handler(SETPRSTATUS+MAP, NULL);
> -	add_handler(UNSETPRSTATUS+MAP, NULL);
> -	add_handler(GETPRKEY+MAP, NULL);
> -	add_handler(SETPRKEY+MAP+KEY, NULL);
> -	add_handler(UNSETPRKEY+MAP, NULL);
> -	add_handler(FORCEQ+DAEMON, NULL);
> -	add_handler(RESTOREQ+DAEMON, NULL);
> -	add_handler(SETMARGINAL+PATH, NULL);
> -	add_handler(UNSETMARGINAL+PATH, NULL);
> -	add_handler(UNSETMARGINAL+MAP, NULL);
> -
>  	return 0;
>  }
>  
> diff --git a/multipathd/cli.h b/multipathd/cli.h
> index 3dac1b4..479a745 100644
> --- a/multipathd/cli.h
> +++ b/multipathd/cli.h
> @@ -133,9 +133,10 @@ struct handler {
>  };
>  
>  int alloc_handlers (void);
> -int add_handler (uint64_t fp, cli_handler *fn);
> -int set_handler_callback (uint64_t fp, cli_handler *fn);
> -int set_unlocked_handler_callback (uint64_t fp, cli_handler *fn);
> +int __set_handler_callback (uint64_t fp, cli_handler *fn, bool locked);
> +#define set_handler_callback(fp, fn) __set_handler_callback(fp, fn, true)
> +#define set_unlocked_handler_callback(fp, fn) __set_handler_callback(fp, fn, false)
> +
>  int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
>  int load_keys (void);
>  char * get_keyparam (vector v, uint64_t code);
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:21PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> This patch sets up the bulk of the state machine. client_state_machine()
> is called in a loop, proceeding from state to state until it needs
> to poll for input or wait for a lock, in which case it returns
> STM_BREAK.
> 
> While doing this, switch to negative error codes for the functions
> in uxlsnr.c (e.g. parse_cmd()). Positive return codes are reserved
> for the cli_handler functions themselves. This way we can clearly
> distinguish the error source, and avoid confusion and misleading
> error messages. No cli_handler returns negative values.
> 
> Note: with this patch applied, clients may hang and time out if
> the handler fails to acquire the vecs lock. This will be fixed in the
> follow-up patch "multipathd: uxlsnr: add idle notification".
> 
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 160 ++++++++++++++++++++++++--------------------
>  1 file changed, 89 insertions(+), 71 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index ff9604f..87134d5 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -299,22 +299,13 @@ static int parse_cmd(struct client *c)
>  
>  	r = get_cmdvec(c->cmd, &c->cmdvec);
>  
> -	if (r) {
> -		genhelp_handler(c->cmd, r, &c->reply);
> -		if (get_strbuf_len(&c->reply) == 0)
> -			return EINVAL;
> -		return 0;
> -	}
> +	if (r)
> +		return -r;
>  
>  	c->handler = find_handler_for_cmdvec(c->cmdvec);
>  
> -	if (!c->handler || !c->handler->fn) {
> -		genhelp_handler(c->cmd, EINVAL, &c->reply);
> -		if (get_strbuf_len(&c->reply) == 0)
> -			r = EINVAL;
> -		else
> -			r = 0;
> -	}
> +	if (!c->handler || !c->handler->fn)
> +		return -EINVAL;
>  
>  	return r;
>  }
> @@ -325,7 +316,7 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
>  	struct timespec tmo;
>  
>  	if (!c->handler)
> -		return EINVAL;
> +		return -EINVAL;
>  
>  	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
>  		tmo.tv_sec += timeout;
> @@ -355,50 +346,30 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
>  	return r;
>  }
>  
> -static int uxsock_trigger(struct client *c, void *trigger_data)
> +void default_reply(struct client *c, int r)
>  {
> -	struct vectors * vecs;
> -	int r = 1;
> -
> -	vecs = (struct vectors *)trigger_data;
> -
> -	r = parse_cmd(c);
> -
> -	if (r == 0 && c->cmdvec && VECTOR_SIZE(c->cmdvec) > 0) {
> -		struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
> -
> -		if (!c->is_root && kw->code != LIST)
> -			r = EPERM;
> -	}
> -
> -	if (r == 0 && c->handler)
> -		r = execute_handler(c, vecs, uxsock_timeout / 1000);
> -
> -	if (c->cmdvec) {
> -		free_keys(c->cmdvec);
> -		c->cmdvec = NULL;
> -	}
> -
> -	if (r > 0) {
> -		switch(r) {
> -		case ETIMEDOUT:
> -			append_strbuf_str(&c->reply, "timeout\n");
> -			break;
> -		case EPERM:
> -			append_strbuf_str(&c->reply,
> -					  "permission deny: need to be root\n");
> -			break;
> -		default:
> -			append_strbuf_str(&c->reply, "fail\n");
> -			break;
> -		}
> -	}
> -	else if (!r && get_strbuf_len(&c->reply) == 0) {
> +	switch(r) {
> +	case -EINVAL:
> +	case -ESRCH:
> +	case -ENOMEM:
> +		/* return codes from get_cmdvec() */
> +		genhelp_handler(c->cmd, -r, &c->reply);
> +		break;
> +	case -EPERM:
> +		append_strbuf_str(&c->reply,
> +				  "permission deny: need to be root\n");
> +		break;
> +	case -ETIMEDOUT:
> +		append_strbuf_str(&c->reply, "timeout\n");
> +		break;
> +	case 0:
>  		append_strbuf_str(&c->reply, "ok\n");
> -		r = 0;
> +		break;
> +	default:
> +		/* cli_handler functions return 1 on unspecified error */
> +		append_strbuf_str(&c->reply, "fail\n");
> +		break;
>  	}
> -	/* else if (r < 0) leave *reply alone */
> -	return r;
>  }
>  
>  static void set_client_state(struct client *c, int state)
> @@ -409,6 +380,7 @@ static void set_client_state(struct client *c, int state)
>  		reset_strbuf(&c->reply);
>  		memset(c->cmd, '\0', sizeof(c->cmd));
>  		c->expires = ts_zero;
> +		c->error = 0;
>  		/* fallthrough */
>  	case CLT_SEND:
>  		/* reuse these fields for next data transfer */
> @@ -420,11 +392,20 @@ static void set_client_state(struct client *c, int state)
>  	c->state = state;
>  }
>  
> -static void handle_client(struct client *c, void *trigger_data)
> +enum {
> +	STM_CONT,
> +	STM_BREAK,
> +};
> +
> +static int client_state_machine(struct client *c, struct vectors *vecs)
>  {
>  	ssize_t n;
> +	const char *buf;
>  
> -	switch (c->state) {
> +	condlog(4, "%s: cli[%d] state=%d cmd=\"%s\" repl \"%s\"", __func__,
> +		c->fd, c->state, c->cmd, get_strbuf_str(&c->reply));
> +
> +        switch (c->state) {
>  	case CLT_RECV:
>  		if (c->cmd_len == 0) {
>  			/*
> @@ -449,31 +430,59 @@ static void handle_client(struct client *c, void *trigger_data)
>  				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
>  			}
>  			/* poll for data */
> -			return;
> +			return STM_BREAK;
>  		} else if (c->len < c->cmd_len) {
>  			n = recv(c->fd, c->cmd + c->len, c->cmd_len - c->len, 0);
>  			if (n <= 0 && errno != EINTR && errno != EAGAIN) {
>  				condlog(1, "%s: cli[%d]: error in recv: %m",
>  					__func__, c->fd);
>  				c->error = -ECONNRESET;
> -				return;
> +				return STM_BREAK;
>  			}
>  			c->len += n;
>  			if (c->len < c->cmd_len)
>  				/* continue polling */
> -				return;
> -			set_client_state(c, CLT_PARSE);
> +				return STM_BREAK;
>  		}
> -		break;
> -	default:
> -		break;
> -	}
> +		condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
> +		set_client_state(c, CLT_PARSE);
> +		return STM_CONT;
>  
> -	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
> -	uxsock_trigger(c, trigger_data);
> +	case CLT_PARSE:
> +		c->error = parse_cmd(c);
> +		if (!c->error) {
> +			/* Permission check */
> +			struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
>  
> -	if (get_strbuf_len(&c->reply) > 0) {
> -		const char *buf = get_strbuf_str(&c->reply);
> +			if (!c->is_root && kw->code != LIST) {
> +				c->error = -EPERM;
> +				condlog(0, "%s: cli[%d]: unauthorized cmd \"%s\"",
> +					__func__, c->fd, c->cmd);
> +			}
> +		}
> +		if (c->error)
> +			set_client_state(c, CLT_SEND);
> +		else
> +			set_client_state(c, CLT_WORK);
> +		return STM_CONT;
> +
> +	case CLT_WAIT_LOCK:
> +		/* tbd */
> +		set_client_state(c, CLT_WORK);
> +		return STM_CONT;
> +
> +	case CLT_WORK:
> +		c->error = execute_handler(c, vecs, uxsock_timeout / 1000);
> +		free_keys(c->cmdvec);
> +		c->cmdvec = NULL;
> +		set_client_state(c, CLT_SEND);
> +		return STM_CONT;
> +
> +	case CLT_SEND:
> +		if (get_strbuf_len(&c->reply) == 0)
> +			default_reply(c, c->error);
> +
> +		buf = get_strbuf_str(&c->reply);
>  
>  		if (send_packet(c->fd, buf) != 0)
>  			dead_client(c);
> @@ -481,9 +490,18 @@ static void handle_client(struct client *c, void *trigger_data)
>  			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
>  				get_strbuf_len(&c->reply) + 1);
>  		reset_strbuf(&c->reply);
> -	}
>  
> -	set_client_state(c, CLT_RECV);
> +		set_client_state(c, CLT_RECV);
> +		return STM_BREAK;
> +
> +	default:
> +		return STM_BREAK;
> +	}
> +}
> +
> +static void handle_client(struct client *c, struct vectors *vecs)
> +{
> +	while (client_state_machine(c, vecs) == STM_CONT);
>  }
>  
>  /*
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:21PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> This patch sets up the bulk of the state machine. client_state_machine()
> is called in a loop, proceeding from state to state until it needs
> to poll for input or wait for a lock, in which case it returns
> STM_BREAK.
> 
> While doing this, switch to negative error codes for the functions
> in uxlsnr.c (e.g. parse_cmd()). Positive return codes are reserved
> for the cli_handler functions themselves. This way we can clearly
> distinguish the error source, and avoid confusion and misleading
> error messages. No cli_handler returns negative values.
> 
> Note: with this patch applied, clients may hang and time out if
> the handler fails to acquire the vecs lock. This will be fixed in the
> follow-up patch "multipathd: uxlsnr: add idle notification".
> 

Actually, one nitpick. See below

> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 160 ++++++++++++++++++++++++--------------------
>  1 file changed, 89 insertions(+), 71 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index ff9604f..87134d5 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -299,22 +299,13 @@ static int parse_cmd(struct client *c)
>  
>  	r = get_cmdvec(c->cmd, &c->cmdvec);
>  
> -	if (r) {
> -		genhelp_handler(c->cmd, r, &c->reply);
> -		if (get_strbuf_len(&c->reply) == 0)
> -			return EINVAL;
> -		return 0;
> -	}
> +	if (r)
> +		return -r;
>  
>  	c->handler = find_handler_for_cmdvec(c->cmdvec);
>  
> -	if (!c->handler || !c->handler->fn) {
> -		genhelp_handler(c->cmd, EINVAL, &c->reply);
> -		if (get_strbuf_len(&c->reply) == 0)
> -			r = EINVAL;
> -		else
> -			r = 0;
> -	}
> +	if (!c->handler || !c->handler->fn)
> +		return -EINVAL;
>  
>  	return r;
>  }
> @@ -325,7 +316,7 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
>  	struct timespec tmo;
>  
>  	if (!c->handler)
> -		return EINVAL;
> +		return -EINVAL;
>  
>  	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
>  		tmo.tv_sec += timeout;
> @@ -355,50 +346,30 @@ static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
>  	return r;
>  }
>  
> -static int uxsock_trigger(struct client *c, void *trigger_data)
> +void default_reply(struct client *c, int r)
>  {
> -	struct vectors * vecs;
> -	int r = 1;
> -
> -	vecs = (struct vectors *)trigger_data;
> -
> -	r = parse_cmd(c);
> -
> -	if (r == 0 && c->cmdvec && VECTOR_SIZE(c->cmdvec) > 0) {
> -		struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
> -
> -		if (!c->is_root && kw->code != LIST)
> -			r = EPERM;
> -	}
> -
> -	if (r == 0 && c->handler)
> -		r = execute_handler(c, vecs, uxsock_timeout / 1000);
> -
> -	if (c->cmdvec) {
> -		free_keys(c->cmdvec);
> -		c->cmdvec = NULL;
> -	}
> -
> -	if (r > 0) {
> -		switch(r) {
> -		case ETIMEDOUT:
> -			append_strbuf_str(&c->reply, "timeout\n");
> -			break;
> -		case EPERM:
> -			append_strbuf_str(&c->reply,
> -					  "permission deny: need to be root\n");
> -			break;
> -		default:
> -			append_strbuf_str(&c->reply, "fail\n");
> -			break;
> -		}
> -	}
> -	else if (!r && get_strbuf_len(&c->reply) == 0) {
> +	switch(r) {
> +	case -EINVAL:
> +	case -ESRCH:
> +	case -ENOMEM:
> +		/* return codes from get_cmdvec() */
> +		genhelp_handler(c->cmd, -r, &c->reply);
> +		break;
> +	case -EPERM:
> +		append_strbuf_str(&c->reply,
> +				  "permission deny: need to be root\n");
> +		break;
> +	case -ETIMEDOUT:
> +		append_strbuf_str(&c->reply, "timeout\n");
> +		break;
> +	case 0:
>  		append_strbuf_str(&c->reply, "ok\n");
> -		r = 0;
> +		break;
> +	default:
> +		/* cli_handler functions return 1 on unspecified error */
> +		append_strbuf_str(&c->reply, "fail\n");
> +		break;
>  	}
> -	/* else if (r < 0) leave *reply alone */
> -	return r;
>  }
>  
>  static void set_client_state(struct client *c, int state)
> @@ -409,6 +380,7 @@ static void set_client_state(struct client *c, int state)
>  		reset_strbuf(&c->reply);
>  		memset(c->cmd, '\0', sizeof(c->cmd));
>  		c->expires = ts_zero;
> +		c->error = 0;
>  		/* fallthrough */
>  	case CLT_SEND:
>  		/* reuse these fields for next data transfer */
> @@ -420,11 +392,20 @@ static void set_client_state(struct client *c, int state)
>  	c->state = state;
>  }
>  
> -static void handle_client(struct client *c, void *trigger_data)
> +enum {
> +	STM_CONT,
> +	STM_BREAK,
> +};
> +
> +static int client_state_machine(struct client *c, struct vectors *vecs)
>  {
>  	ssize_t n;
> +	const char *buf;
>  
> -	switch (c->state) {
> +	condlog(4, "%s: cli[%d] state=%d cmd=\"%s\" repl \"%s\"", __func__,
> +		c->fd, c->state, c->cmd, get_strbuf_str(&c->reply));
> +

This switch statement is indented with 8 spaces, instead of a tab

-Ben

> +        switch (c->state) {
>  	case CLT_RECV:
>  		if (c->cmd_len == 0) {
>  			/*
> @@ -449,31 +430,59 @@ static void handle_client(struct client *c, void *trigger_data)
>  				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
>  			}
>  			/* poll for data */
> -			return;
> +			return STM_BREAK;
>  		} else if (c->len < c->cmd_len) {
>  			n = recv(c->fd, c->cmd + c->len, c->cmd_len - c->len, 0);
>  			if (n <= 0 && errno != EINTR && errno != EAGAIN) {
>  				condlog(1, "%s: cli[%d]: error in recv: %m",
>  					__func__, c->fd);
>  				c->error = -ECONNRESET;
> -				return;
> +				return STM_BREAK;
>  			}
>  			c->len += n;
>  			if (c->len < c->cmd_len)
>  				/* continue polling */
> -				return;
> -			set_client_state(c, CLT_PARSE);
> +				return STM_BREAK;
>  		}
> -		break;
> -	default:
> -		break;
> -	}
> +		condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
> +		set_client_state(c, CLT_PARSE);
> +		return STM_CONT;
>  
> -	condlog(4, "cli[%d]: Got request [%s]", c->fd, c->cmd);
> -	uxsock_trigger(c, trigger_data);
> +	case CLT_PARSE:
> +		c->error = parse_cmd(c);
> +		if (!c->error) {
> +			/* Permission check */
> +			struct key *kw = VECTOR_SLOT(c->cmdvec, 0);
>  
> -	if (get_strbuf_len(&c->reply) > 0) {
> -		const char *buf = get_strbuf_str(&c->reply);
> +			if (!c->is_root && kw->code != LIST) {
> +				c->error = -EPERM;
> +				condlog(0, "%s: cli[%d]: unauthorized cmd \"%s\"",
> +					__func__, c->fd, c->cmd);
> +			}
> +		}
> +		if (c->error)
> +			set_client_state(c, CLT_SEND);
> +		else
> +			set_client_state(c, CLT_WORK);
> +		return STM_CONT;
> +
> +	case CLT_WAIT_LOCK:
> +		/* tbd */
> +		set_client_state(c, CLT_WORK);
> +		return STM_CONT;
> +
> +	case CLT_WORK:
> +		c->error = execute_handler(c, vecs, uxsock_timeout / 1000);
> +		free_keys(c->cmdvec);
> +		c->cmdvec = NULL;
> +		set_client_state(c, CLT_SEND);
> +		return STM_CONT;
> +
> +	case CLT_SEND:
> +		if (get_strbuf_len(&c->reply) == 0)
> +			default_reply(c, c->error);
> +
> +		buf = get_strbuf_str(&c->reply);
>  
>  		if (send_packet(c->fd, buf) != 0)
>  			dead_client(c);
> @@ -481,9 +490,18 @@ static void handle_client(struct client *c, void *trigger_data)
>  			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
>  				get_strbuf_len(&c->reply) + 1);
>  		reset_strbuf(&c->reply);
> -	}
>  
> -	set_client_state(c, CLT_RECV);
> +		set_client_state(c, CLT_RECV);
> +		return STM_BREAK;
> +
> +	default:
> +		return STM_BREAK;
> +	}
> +}
> +
> +static void handle_client(struct client *c, struct vectors *vecs)
> +{
> +	while (client_state_machine(c, vecs) == STM_CONT);
>  }
>  
>  /*
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 30/48] multipathd: uxlsnr: add idle notification

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:22PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> The previous patches added the state machine and the timeout handling,
> but there was no wakeup mechanism for the uxlsnr for cases where
> client connections were waiting for the vecs lock.
> 
> This patch uses the previously introduced wakeup mechanism of
> struct mutex_lock for this purpose. Processes which unlock the
> "global" vecs lock send an event in an eventfd which the uxlsnr
> loop is polling for.
> 
> As we are now woken up for servicing client handlers that don't
> wait for input but for the lock, we need to set up the pollfds
> differently, and iterate over all clients when handling events,
> not only over the ones that are receiving. The hangup handling
> is changed, too. We have to look at every client, even if one has
> hung up. Note that I don't take client_lock for the loop in
> uxsock_listen(), it's not necessary and will be removed elsewhere
> in a follow-up patch.
> 
> With this in place, the lock need not be taken in execute_handler()
> any more. The uxlsnr only ever calls trylock() on the vecs lock,
> avoiding any waiting for other threads to finish.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 200 ++++++++++++++++++++++++++++----------------
>  1 file changed, 129 insertions(+), 71 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index 87134d5..bf9780d 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -24,6 +24,7 @@
>  #include <signal.h>
>  #include <stdbool.h>
>  #include <sys/inotify.h>
> +#include <sys/eventfd.h>
>  #include "checkers.h"
>  #include "memory.h"
>  #include "debug.h"
> @@ -70,6 +71,7 @@ struct client {
>  enum {
>  	POLLFD_UX = 0,
>  	POLLFD_NOTIFY,
> +	POLLFD_IDLE,
>  	POLLFDS_BASE,
>  };
>  
> @@ -90,6 +92,7 @@ static LIST_HEAD(clients);
>  static pthread_mutex_t client_lock = PTHREAD_MUTEX_INITIALIZER;
>  static struct pollfd *polls;
>  static int notify_fd = -1;
> +static int idle_fd = -1;
>  static char *watch_config_dir;
>  
>  static bool _socket_client_is_root(int fd)
> @@ -187,6 +190,17 @@ void uxsock_cleanup(void *arg)
>  	free_polls();
>  }
>  
> +void wakeup_cleanup(void *arg)
> +{
> +	struct mutex_lock *lck = arg;
> +	int fd = idle_fd;
> +
> +	idle_fd = -1;
> +	set_wakeup_fn(lck, NULL);
> +	if (fd != -1)
> +		close(fd);
> +}
> +
>  struct watch_descriptors {
>  	int conf_wd;
>  	int dir_wd;
> @@ -293,6 +307,18 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
>  
>  static const struct timespec ts_zero = { .tv_sec = 0, };
>  
> +/* call with clients lock held */
> +static bool __need_vecs_lock(void)
> +{
> +	struct client *c;
> +
> +	list_for_each_entry(c, &clients, node) {
> +		if (c->state == CLT_WAIT_LOCK)
> +			return true;
> +	}
> +	return false;
> +}
> +
>  static int parse_cmd(struct client *c)
>  {
>  	int r;
> @@ -310,40 +336,31 @@ static int parse_cmd(struct client *c)
>  	return r;
>  }
>  
> -static int execute_handler(struct client *c, struct vectors *vecs, int timeout)
> +static int execute_handler(struct client *c, struct vectors *vecs)
>  {
> -	int r;
> -	struct timespec tmo;
>  
> -	if (!c->handler)
> +	if (!c->handler || !c->handler->fn)
>  		return -EINVAL;
>  
> -	if (clock_gettime(CLOCK_REALTIME, &tmo) == 0) {
> -		tmo.tv_sec += timeout;
> -	} else {
> -		tmo.tv_sec = 0;
> -	}
> +	return c->handler->fn(c->cmdvec, &c->reply, vecs);
> +}
>  
> -	if (c->handler->locked) {
> -		int locked = 0;
> +static void wakeup_listener(void)
> +{
> +	uint64_t one = 1;
>  
> -		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> -		if (tmo.tv_sec) {
> -			r = timedlock(&vecs->lock, &tmo);
> -		} else {
> -			lock(&vecs->lock);
> -			r = 0;
> -		}
> -		if (r == 0) {
> -			locked = 1;
> -			pthread_testcancel();
> -			r = c->handler->fn(c->cmdvec, &c->reply, vecs);
> -		}
> -		pthread_cleanup_pop(locked);
> -	} else
> -		r = c->handler->fn(c->cmdvec, &c->reply, vecs);
> +	if (idle_fd != -1 &&
> +	    write(idle_fd, &one, sizeof(one)) != sizeof(one))
> +		condlog(1, "%s: failed", __func__);
> +}
>  
> -	return r;
> +static void drain_idle_fd(int fd)
> +{
> +	uint64_t val;
> +	int rc;
> +
> +	rc = read(fd, &val, sizeof(val));
> +	condlog(4, "%s: %d, %"PRIu64, __func__, rc, val);
>  }
>  
>  void default_reply(struct client *c, int r)
> @@ -397,16 +414,19 @@ enum {
>  	STM_BREAK,
>  };
>  
> -static int client_state_machine(struct client *c, struct vectors *vecs)
> +static int client_state_machine(struct client *c, struct vectors *vecs,
> +				short revents)
>  {
>  	ssize_t n;
>  	const char *buf;
>  
> -	condlog(4, "%s: cli[%d] state=%d cmd=\"%s\" repl \"%s\"", __func__,
> -		c->fd, c->state, c->cmd, get_strbuf_str(&c->reply));
> +	condlog(4, "%s: cli[%d] poll=%x state=%d cmd=\"%s\" repl \"%s\"", __func__,
> +		c->fd, revents, c->state, c->cmd, get_strbuf_str(&c->reply));
>  
>          switch (c->state) {
>  	case CLT_RECV:
> +		if (!(revents & POLLIN))
> +			return STM_BREAK;
>  		if (c->cmd_len == 0) {
>  			/*
>  			 * We got POLLIN; assume that at least the length can
> @@ -462,17 +482,30 @@ static int client_state_machine(struct client *c, struct vectors *vecs)
>  		}
>  		if (c->error)
>  			set_client_state(c, CLT_SEND);
> +		else if (c->handler->locked)
> +			set_client_state(c, CLT_WAIT_LOCK);
>  		else
>  			set_client_state(c, CLT_WORK);
>  		return STM_CONT;
>  
>  	case CLT_WAIT_LOCK:

It's not a big deal, but I would prefer a name like CLT_LOCKED_WORK
instead of CLT_WAIT_LOCK, to make it obvious that these are alternate
possibilites.


> -		/* tbd */
> -		set_client_state(c, CLT_WORK);
> -		return STM_CONT;
> +                if (trylock(&vecs->lock) == 0) {
> +			/* don't use cleanup_lock(), lest we wakeup ourselves */
> +			pthread_cleanup_push_cast(__unlock, &vecs->lock);
> +			c->error = execute_handler(c, vecs);
> +			pthread_cleanup_pop(1);
> +			condlog(4, "%s: cli[%d] grabbed lock", __func__, c->fd);
> +			free_keys(c->cmdvec);
> +			c->cmdvec = NULL;
> +			set_client_state(c, CLT_SEND);
> +			return STM_CONT;
> +		} else {
> +			condlog(4, "%s: cli[%d] waiting for lock", __func__, c->fd);
> +			return STM_BREAK;
> +		}
>  
>  	case CLT_WORK:
> -		c->error = execute_handler(c, vecs, uxsock_timeout / 1000);
> +		c->error = execute_handler(c, vecs);
>  		free_keys(c->cmdvec);
>  		c->cmdvec = NULL;
>  		set_client_state(c, CLT_SEND);
> @@ -499,9 +532,14 @@ static int client_state_machine(struct client *c, struct vectors *vecs)
>  	}
>  }
>  
> -static void handle_client(struct client *c, struct vectors *vecs)
> +static void handle_client(struct client *c, struct vectors *vecs, short revents)
>  {
> -	while (client_state_machine(c, vecs) == STM_CONT);
> +	if (revents & (POLLHUP|POLLERR)) {
> +		c->error = -ECONNRESET;
> +		return;
> +	}
> +
> +        while (client_state_machine(c, vecs, revents) == STM_CONT);
>  }
>  
>  /*
> @@ -514,6 +552,8 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  	/* conf->sequence_nr will be 1 when uxsock_listen is first called */
>  	unsigned int sequence_nr = 0;
>  	struct watch_descriptors wds = { .conf_wd = -1, .dir_wd = -1 };
> +	bool need_lock = false;
> +	struct vectors *vecs = trigger_data;
>  
>  	condlog(3, "uxsock: startup listener");
>  	polls = MALLOC(max_pfds * sizeof(*polls));
> @@ -524,6 +564,14 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  	notify_fd = inotify_init1(IN_NONBLOCK);
>  	if (notify_fd == -1) /* it's fine if notifications fail */
>  		condlog(3, "failed to start up configuration notifications");
> +
> +	pthread_cleanup_push(wakeup_cleanup, &vecs->lock);
> +	idle_fd = eventfd(0, EFD_NONBLOCK|EFD_CLOEXEC);
> +	if (idle_fd == -1)
> +		condlog(1, "failed to create idle fd");

If the idle_fd doesn't get correctly set, that seems like a fatal error
to me.

> +	else
> +		set_wakeup_fn(&vecs->lock, wakeup_listener);
> +
>  	sigfillset(&mask);
>  	sigdelset(&mask, SIGINT);
>  	sigdelset(&mask, SIGTERM);
> @@ -575,16 +623,30 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  		else
>  			polls[POLLFD_NOTIFY].events = POLLIN;
>  
> +		need_lock = __need_vecs_lock();
> +		polls[POLLFD_IDLE].fd = idle_fd;
> +		if (need_lock)
> +			polls[POLLFD_IDLE].events = POLLIN;
> +		else
> +			polls[POLLFD_IDLE].events = 0;
> +
>  		/* setup the clients */
> -		i = POLLFDS_BASE;
> -		list_for_each_entry(c, &clients, node) {
> -			polls[i].fd = c->fd;
> -			polls[i].events = POLLIN;
> -			i++;
> -			if (i >= max_pfds)
> -				break;
> -		}
> -		n_pfds = i;
> +                i = POLLFDS_BASE;
> +                list_for_each_entry(c, &clients, node) {

Nitpick: This would look clearer to me if, instead of a switch
statement, it was just

if (c->state != CLT_RECV)
        continue;

polls[i].events = POLLIN;
polls[i].fd = c->fd;
...


-Ben

> +                        switch(c->state) {
> +                        case CLT_RECV:
> +                                polls[i].events = POLLIN;
> +                                break;
> +                        default:
> +				/* don't poll for this client */
> +                                continue;
> +                        }
> +                        polls[i].fd = c->fd;
> +                        i++;
> +                        if (i >= max_pfds)
> +                                break;
> +                }
> +                n_pfds = i;
>  		pthread_cleanup_pop(1);
>  
>  		/* most of our life is spent in this call */
> @@ -607,33 +669,28 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  			handle_signals(true);
>  			continue;
>  		}
> +		if (polls[POLLFD_IDLE].fd != -1 &&
> +		    polls[POLLFD_IDLE].revents & POLLIN)
> +			drain_idle_fd(idle_fd);
>  
> -		/* see if a client wants to speak to us */
> -		for (i = POLLFDS_BASE; i < n_pfds; i++) {
> -			if (polls[i].revents & (POLLIN|POLLHUP|POLLERR)) {
> -				c = NULL;
> -				pthread_mutex_lock(&client_lock);
> -				list_for_each_entry(tmp, &clients, node) {
> -					if (tmp->fd == polls[i].fd) {
> -						c = tmp;
> -						break;
> -					}
> -				}
> -				pthread_mutex_unlock(&client_lock);
> -				if (!c) {
> -					condlog(4, "cli%d: new fd %d",
> -						i, polls[i].fd);
> -					continue;
> -				}
> -				if (polls[i].revents & (POLLHUP|POLLERR)) {
> -					condlog(4, "cli[%d]: Disconnected",
> -						c->fd);
> -					dead_client(c);
> -					continue;
> -				}
> -				handle_client(c, trigger_data);
> -				if (c->error == -ECONNRESET)
> -					dead_client(c);
> +		/* see if a client needs handling */
> +		list_for_each_entry_safe(c, tmp, &clients, node) {
> +			short revents = 0;
> +
> +			for (i = POLLFDS_BASE; i < n_pfds; i++) {
> +                                if (polls[i].fd == c->fd) {
> +                                        revents = polls[i].revents;
> +                                        break;
> +                                }
> +                        }
> +
> +			handle_client(c, trigger_data, revents);
> +
> +			if (c->error == -ECONNRESET) {
> +				condlog(4, "cli[%d]: disconnected", c->fd);
> +				dead_client(c);
> +				if (i < n_pfds)
> +					polls[i].fd = -1;
>  			}
>  		}
>  		/* see if we got a non-fatal signal */
> @@ -649,5 +706,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  			handle_inotify(notify_fd, &wds);
>  	}
>  
> +	pthread_cleanup_pop(1);
>  	return NULL;
>  }
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 31/48] multipathd: uxlsnr: add timeout handling

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:23PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> Our ppoll() call needs to wake up when a client request times out.
> This logic can be added by determining the first client that's about
> to time out. The logic in handle_client() will then cause a timeout
> reply to be sent to the client. This is more client-friendly
> as the client timing out without receiving a reply.
> 
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 57 +++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 52 insertions(+), 5 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index bf9780d..45fe7b5 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -306,6 +306,35 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
>  }
>  
>  static const struct timespec ts_zero = { .tv_sec = 0, };
> +static const struct timespec ts_max = { .tv_sec = LONG_MAX, .tv_nsec = 999999999 };
> +
> +/* call with clients lock held */
> +static struct timespec *__get_soonest_timeout(struct timespec *ts)
> +{
> +	struct timespec ts_min = ts_max, now;
> +	bool any = false;
> +	struct client *c;
> +
> +	list_for_each_entry(c, &clients, node) {
> +		if (timespeccmp(&c->expires, &ts_zero) != 0 &&
> +		    timespeccmp(&c->expires, &ts_min) < 0) {
> +			ts_min = c->expires;
> +			any = true;
> +		}
> +	}
> +
> +	if (!any)
> +		return NULL;
> +
> +	get_monotonic_time(&now);
> +	timespecsub(&ts_min, &now, ts);
> +	if (timespeccmp(ts, &ts_zero) < 0)
> +		*ts = ts_zero;
> +
> +	condlog(4, "%s: next client expires in %ld.%03lds", __func__,
> +		(long)ts->tv_sec, ts->tv_nsec / 1000000);
> +	return ts;
> +}
>  
>  /* call with clients lock held */
>  static bool __need_vecs_lock(void)
> @@ -532,6 +561,24 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
>  	}
>  }
>  
> +static void check_timeout(struct client *c)
> +{
> +	struct timespec now;
> +
> +	if (timespeccmp(&c->expires, &ts_zero) == 0)
> +		return;
> +
> +	get_monotonic_time(&now);
> +	if (timespeccmp(&c->expires, &now) > 0)
> +		return;
> +
> +	condlog(2, "%s: cli[%d]: timed out at %ld.%03ld", __func__,
> +		c->fd, (long)c->expires.tv_sec, c->expires.tv_nsec / 1000000);
> +
> +	c->error = -ETIMEDOUT;
> +	set_client_state(c, CLT_SEND);
> +}
> +
>  static void handle_client(struct client *c, struct vectors *vecs, short revents)
>  {
>  	if (revents & (POLLHUP|POLLERR)) {
> @@ -539,6 +586,7 @@ static void handle_client(struct client *c, struct vectors *vecs, short revents)
>  		return;
>  	}
>  
> +	check_timeout(c);
>          while (client_state_machine(c, vecs, revents) == STM_CONT);
>  }
>  
> @@ -580,6 +628,7 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  	while (1) {
>  		struct client *c, *tmp;
>  		int i, n_pfds, poll_count, num_clients;
> +		struct timespec __timeout, *timeout;
>  
>  		/* setup for a poll */
>  		pthread_mutex_lock(&client_lock);
> @@ -647,10 +696,12 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>                                  break;
>                  }
>                  n_pfds = i;
> +		timeout = __get_soonest_timeout(&__timeout);
> +
>  		pthread_cleanup_pop(1);
>  
>  		/* most of our life is spent in this call */
> -		poll_count = ppoll(polls, n_pfds, NULL, &mask);
> +		poll_count = ppoll(polls, n_pfds, timeout, &mask);
>  
>  		handle_signals(false);
>  		if (poll_count == -1) {
> @@ -665,10 +716,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  			break;
>  		}
>  
> -		if (poll_count == 0) {
> -			handle_signals(true);
> -			continue;
> -		}
>  		if (polls[POLLFD_IDLE].fd != -1 &&
>  		    polls[POLLFD_IDLE].revents & POLLIN)
>  			drain_idle_fd(idle_fd);
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 32/48] multipathd: uxlsnr: use poll loop for sending, too

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:24PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> send_packet() may busy-loop. By polling for POLLOUT, we can
> avoid that, even if it's very unlikely in practice.
> 

The last time I reviewed this, I mentioned that when we fall through
from CLT_WORK to CLT_SEND, the client hasn't checked for a POLLOUT
revent, so it's possible to block here. I'm not sure that we care.
If not

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>

> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 35 ++++++++++++++++++++++++++---------
>  1 file changed, 26 insertions(+), 9 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index 45fe7b5..6213454 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -447,7 +447,6 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
>  				short revents)
>  {
>  	ssize_t n;
> -	const char *buf;
>  
>  	condlog(4, "%s: cli[%d] poll=%x state=%d cmd=\"%s\" repl \"%s\"", __func__,
>  		c->fd, revents, c->state, c->cmd, get_strbuf_str(&c->reply));
> @@ -544,16 +543,31 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
>  		if (get_strbuf_len(&c->reply) == 0)
>  			default_reply(c, c->error);
>  
> -		buf = get_strbuf_str(&c->reply);
> +		if (c->cmd_len == 0) {
> +			size_t len = get_strbuf_len(&c->reply) + 1;
>  
> -		if (send_packet(c->fd, buf) != 0)
> -			dead_client(c);
> -		else
> -			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd,
> -				get_strbuf_len(&c->reply) + 1);
> -		reset_strbuf(&c->reply);
> +			if (send(c->fd, &len, sizeof(len), MSG_NOSIGNAL)
> +			    != sizeof(len))
> +				c->error = -ECONNRESET;
> +			c->cmd_len = len;
> +			return STM_BREAK;
> +		}
>  
> -		set_client_state(c, CLT_RECV);
> +		if (c->len < c->cmd_len) {
> +			const char *buf = get_strbuf_str(&c->reply);
> +
> +			n = send(c->fd, buf + c->len, c->cmd_len, MSG_NOSIGNAL);
> +			if (n == -1) {
> +				if (!(errno == EAGAIN || errno == EINTR))
> +					c->error = -ECONNRESET;
> +			} else
> +				c->len += n;
> +		}
> +
> +                if (c->len >= c->cmd_len) {
> +			condlog(4, "cli[%d]: Reply [%zu bytes]", c->fd, c->cmd_len);
> +			set_client_state(c, CLT_RECV);
> +		}
>  		return STM_BREAK;
>  
>  	default:
> @@ -686,6 +700,9 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>                          case CLT_RECV:
>                                  polls[i].events = POLLIN;
>                                  break;
> +			case CLT_SEND:
> +				polls[i].events = POLLOUT;
> +				break;
>                          default:
>  				/* don't poll for this client */
>                                  continue;
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 33/48] multipathd: uxlsnr: drop client_lock

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:25PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> The list of clients is never changed anywhere except in
> uxsock_listen(). No need to lock.
> 
> Signed-off-by: Martin Wilck <mwilck@suse.com>
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> ---
>  multipathd/uxlsnr.c | 33 ++++++---------------------------
>  1 file changed, 6 insertions(+), 27 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index 6213454..24db377 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -89,7 +89,6 @@ enum {
>  static __attribute__((unused)) char ___a[-(MIN_POLLS <= 0)];
>  
>  static LIST_HEAD(clients);
> -static pthread_mutex_t client_lock = PTHREAD_MUTEX_INITIALIZER;
>  static struct pollfd *polls;
>  static int notify_fd = -1;
>  static int idle_fd = -1;
> @@ -136,15 +135,13 @@ static void new_client(int ux_sock)
>  	c->is_root = _socket_client_is_root(c->fd);
>  
>  	/* put it in our linked list */
> -	pthread_mutex_lock(&client_lock);
>  	list_add_tail(&c->node, &clients);
> -	pthread_mutex_unlock(&client_lock);
>  }
>  
>  /*
>   * kill off a dead client
>   */
> -static void _dead_client(struct client *c)
> +static void dead_client(struct client *c)
>  {
>  	int fd = c->fd;
>  	list_del_init(&c->node);
> @@ -156,14 +153,6 @@ static void _dead_client(struct client *c)
>  	close(fd);
>  }
>  
> -static void dead_client(struct client *c)
> -{
> -	pthread_cleanup_push(cleanup_mutex, &client_lock);
> -	pthread_mutex_lock(&client_lock);
> -	_dead_client(c);
> -	pthread_cleanup_pop(1);
> -}
> -
>  static void free_polls (void)
>  {
>  	if (polls)
> @@ -180,11 +169,9 @@ void uxsock_cleanup(void *arg)
>  	close(notify_fd);
>  	free(watch_config_dir);
>  
> -	pthread_mutex_lock(&client_lock);
>  	list_for_each_entry_safe(client_loop, client_tmp, &clients, node) {
> -		_dead_client(client_loop);
> +		dead_client(client_loop);
>  	}
> -	pthread_mutex_unlock(&client_lock);
>  
>  	cli_exit();
>  	free_polls();
> @@ -308,8 +295,7 @@ static void handle_inotify(int fd, struct watch_descriptors *wds)
>  static const struct timespec ts_zero = { .tv_sec = 0, };
>  static const struct timespec ts_max = { .tv_sec = LONG_MAX, .tv_nsec = 999999999 };
>  
> -/* call with clients lock held */
> -static struct timespec *__get_soonest_timeout(struct timespec *ts)
> +static struct timespec *get_soonest_timeout(struct timespec *ts)
>  {
>  	struct timespec ts_min = ts_max, now;
>  	bool any = false;
> @@ -336,8 +322,7 @@ static struct timespec *__get_soonest_timeout(struct timespec *ts)
>  	return ts;
>  }
>  
> -/* call with clients lock held */
> -static bool __need_vecs_lock(void)
> +static bool need_vecs_lock(void)
>  {
>  	struct client *c;
>  
> @@ -614,7 +599,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  	/* conf->sequence_nr will be 1 when uxsock_listen is first called */
>  	unsigned int sequence_nr = 0;
>  	struct watch_descriptors wds = { .conf_wd = -1, .dir_wd = -1 };
> -	bool need_lock = false;
>  	struct vectors *vecs = trigger_data;
>  
>  	condlog(3, "uxsock: startup listener");
> @@ -645,8 +629,6 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  		struct timespec __timeout, *timeout;
>  
>  		/* setup for a poll */
> -		pthread_mutex_lock(&client_lock);
> -		pthread_cleanup_push(cleanup_mutex, &client_lock);
>  		num_clients = 0;
>  		list_for_each_entry(c, &clients, node) {
>  			num_clients++;
> @@ -686,9 +668,8 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>  		else
>  			polls[POLLFD_NOTIFY].events = POLLIN;
>  
> -		need_lock = __need_vecs_lock();
>  		polls[POLLFD_IDLE].fd = idle_fd;
> -		if (need_lock)
> +		if (need_vecs_lock())
>  			polls[POLLFD_IDLE].events = POLLIN;
>  		else
>  			polls[POLLFD_IDLE].events = 0;
> @@ -713,9 +694,7 @@ void *uxsock_listen(long ux_sock, void *trigger_data)
>                                  break;
>                  }
>                  n_pfds = i;
> -		timeout = __get_soonest_timeout(&__timeout);
> -
> -		pthread_cleanup_pop(1);
> +		timeout = get_soonest_timeout(&__timeout);
>  
>  		/* most of our life is spent in this call */
>  		poll_count = ppoll(polls, n_pfds, timeout, &mask);
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 35/48] multipathd: uxlsnr: use recv() for command length

[Benjamin Marzinski]

On Thu, Nov 18, 2021 at 11:58:27PM +0100, mwilck@suse.com wrote:
> From: Martin Wilck <mwilck@suse.com>
> 
> If the peer uses libmpathcmd, we can be certain that the first
> 8 bytes are being sent in a single chunk of data. It's overkill
> to try and receive the command length byte-by-byte.
> 
Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
>  multipathd/uxlsnr.c | 17 +++++++++--------
>  1 file changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> index 24db377..6355279 100644
> --- a/multipathd/uxlsnr.c
> +++ b/multipathd/uxlsnr.c
> @@ -441,6 +441,7 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
>  		if (!(revents & POLLIN))
>  			return STM_BREAK;
>  		if (c->cmd_len == 0) {
> +			size_t len;
>  			/*
>  			 * We got POLLIN; assume that at least the length can
>  			 * be read immediately.
> @@ -449,17 +450,17 @@ static int client_state_machine(struct client *c, struct vectors *vecs,
>  			c->expires.tv_sec += uxsock_timeout / 1000;
>  			c->expires.tv_nsec += (uxsock_timeout % 1000) * 1000000;
>  			normalize_timespec(&c->expires);
> -			n = mpath_recv_reply_len(c->fd, 0);
> -			if (n == -1) {
> -				condlog(1, "%s: cli[%d]: failed to receive reply len",
> -					__func__, c->fd);
> -				c->error = -ECONNRESET;
> -			} else if (n > _MAX_CMD_LEN) {
> -				condlog(1, "%s: cli[%d]: overlong command (%zd bytes)",
> +			n = recv(c->fd, &len, sizeof(len), 0);
> +			if (n < (ssize_t)sizeof(len)) {
> +				condlog(1, "%s: cli[%d]: failed to receive reply len: %zd",
>  					__func__, c->fd, n);
>  				c->error = -ECONNRESET;
> +			} else if (len <= 0 || len > _MAX_CMD_LEN) {
> +				condlog(1, "%s: cli[%d]: invalid command length (%zu bytes)",
> +					__func__, c->fd, len);
> +				c->error = -ECONNRESET;
>  			} else {
> -				c->cmd_len = n;
> +				c->cmd_len = len;
>  				condlog(4, "%s: cli[%d]: connected", __func__, c->fd);
>  			}
>  			/* poll for data */
> -- 
> 2.33.1

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 32/48] multipathd: uxlsnr: use poll loop for sending, too

[Martin Wilck]

On Wed, 2021-11-24 at 19:43 -0600, Benjamin Marzinski wrote:
> On Thu, Nov 18, 2021 at 11:58:24PM +0100, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > send_packet() may busy-loop. By polling for POLLOUT, we can
> > avoid that, even if it's very unlikely in practice.
> > 
> 
> The last time I reviewed this, I mentioned that when we fall through
> from CLT_WORK to CLT_SEND, the client hasn't checked for a POLLOUT
> revent, so it's possible to block here. I'm not sure that we care.

Fixing that will be as easy as entering as returning once from the
state machine. I'll add the fix.

Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 30/48] multipathd: uxlsnr: add idle notification

[Martin Wilck]

On Wed, 2021-11-24 at 19:08 -0600, Benjamin Marzinski wrote:
> On Thu, Nov 18, 2021 at 11:58:22PM +0100, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > The previous patches added the state machine and the timeout
> > handling,
> > but there was no wakeup mechanism for the uxlsnr for cases where
> > client connections were waiting for the vecs lock.
> > 
> > This patch uses the previously introduced wakeup mechanism of
> > struct mutex_lock for this purpose. Processes which unlock the
> > "global" vecs lock send an event in an eventfd which the uxlsnr
> > loop is polling for.
> > 
> > As we are now woken up for servicing client handlers that don't
> > wait for input but for the lock, we need to set up the pollfds
> > differently, and iterate over all clients when handling events,
> > not only over the ones that are receiving. The hangup handling
> > is changed, too. We have to look at every client, even if one has
> > hung up. Note that I don't take client_lock for the loop in
> > uxsock_listen(), it's not necessary and will be removed elsewhere
> > in a follow-up patch.
> > 
> > With this in place, the lock need not be taken in execute_handler()
> > any more. The uxlsnr only ever calls trylock() on the vecs lock,
> > avoiding any waiting for other threads to finish.
> > 
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > ---
> >  multipathd/uxlsnr.c | 200 ++++++++++++++++++++++++++++------------
> > ----
> >  1 file changed, 129 insertions(+), 71 deletions(-)
> > 
> > diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> > index 87134d5..bf9780d 100644
> > --- a/multipathd/uxlsnr.c
> > +++ b/multipathd/uxlsnr.c
> 
> Nitpick: This would look clearer to me if, instead of a switch
> statement, it was just
> 
> if (c->state != CLT_RECV)
>         continue;
> 
> polls[i].events = POLLIN;
> polls[i].fd = c->fd;
> ...

That's true if you look at this patch in isolation. The reason I use a
switch statement is that with patch 32, we get another case to treat
here (CLT_SEND). At that point, the switch is at least on par wrt
clarity, IMO.

No?

Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[Martin Wilck]

On Wed, 2021-11-24 at 18:38 -0600, Benjamin Marzinski wrote:
> On Thu, Nov 18, 2021 at 11:58:21PM +0100, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > This patch sets up the bulk of the state machine.
> > client_state_machine()
> > is called in a loop, proceeding from state to state until it needs
> > to poll for input or wait for a lock, in which case it returns
> > STM_BREAK.
> > 
> > While doing this, switch to negative error codes for the functions
> > in uxlsnr.c (e.g. parse_cmd()). Positive return codes are reserved
> > for the cli_handler functions themselves. This way we can clearly
> > distinguish the error source, and avoid confusion and misleading
> > error messages. No cli_handler returns negative values.
> > 
> > Note: with this patch applied, clients may hang and time out if
> > the handler fails to acquire the vecs lock. This will be fixed in
> > the
> > follow-up patch "multipathd: uxlsnr: add idle notification".
> > 
> 
> Actually, one nitpick. See below
> 
> > +
> 
> This switch statement is indented with 8 spaces, instead of a tab

I'm going to fix that, but I assume you're aware that our code is far
from being consistent in this respect. This holds also for other
patches in this series. Do you want me to re-format all of them?

Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[Benjamin Marzinski]

On Fri, Nov 26, 2021 at 03:34:59PM +0100, Martin Wilck wrote:
> On Wed, 2021-11-24 at 18:38 -0600, Benjamin Marzinski wrote:
> > On Thu, Nov 18, 2021 at 11:58:21PM +0100, mwilck@suse.com wrote:
> > > From: Martin Wilck <mwilck@suse.com>
> > > 
> > > This patch sets up the bulk of the state machine.
> > > client_state_machine()
> > > is called in a loop, proceeding from state to state until it needs
> > > to poll for input or wait for a lock, in which case it returns
> > > STM_BREAK.
> > > 
> > > While doing this, switch to negative error codes for the functions
> > > in uxlsnr.c (e.g. parse_cmd()). Positive return codes are reserved
> > > for the cli_handler functions themselves. This way we can clearly
> > > distinguish the error source, and avoid confusion and misleading
> > > error messages. No cli_handler returns negative values.
> > > 
> > > Note: with this patch applied, clients may hang and time out if
> > > the handler fails to acquire the vecs lock. This will be fixed in
> > > the
> > > follow-up patch "multipathd: uxlsnr: add idle notification".
> > > 
> > 
> > Actually, one nitpick. See below
> > 
> > > +
> > 
> > This switch statement is indented with 8 spaces, instead of a tab
> 
> I'm going to fix that, but I assume you're aware that our code is far
> from being consistent in this respect. This holds also for other
> patches in this series. Do you want me to re-format all of them?

Huh? I mean that the line doesn't start with a tab, but instead has 8
spaces.  A quick grep through the source code in your queue branch only
turns that up in some of the files in the tests directory and in files
we've just imported

# grep -l "^        " `find ./ -name "*.[ch]"`
./libmultipath/nvme/nvme-ioctl.c
./tests/pgpolicy.c
./tests/util.c
./tests/directio.c
./tests/mpathvalid.c
./tests/dmevents.c
./third-party/valgrind/drd.h
./third-party/valgrind/valgrind.h

-Ben

> 
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 30/48] multipathd: uxlsnr: add idle notification

[Benjamin Marzinski]

On Fri, Nov 26, 2021 at 03:23:18PM +0100, Martin Wilck wrote:
> On Wed, 2021-11-24 at 19:08 -0600, Benjamin Marzinski wrote:
> > On Thu, Nov 18, 2021 at 11:58:22PM +0100, mwilck@suse.com wrote:
> > > From: Martin Wilck <mwilck@suse.com>
> > > 
> > > The previous patches added the state machine and the timeout
> > > handling,
> > > but there was no wakeup mechanism for the uxlsnr for cases where
> > > client connections were waiting for the vecs lock.
> > > 
> > > This patch uses the previously introduced wakeup mechanism of
> > > struct mutex_lock for this purpose. Processes which unlock the
> > > "global" vecs lock send an event in an eventfd which the uxlsnr
> > > loop is polling for.
> > > 
> > > As we are now woken up for servicing client handlers that don't
> > > wait for input but for the lock, we need to set up the pollfds
> > > differently, and iterate over all clients when handling events,
> > > not only over the ones that are receiving. The hangup handling
> > > is changed, too. We have to look at every client, even if one has
> > > hung up. Note that I don't take client_lock for the loop in
> > > uxsock_listen(), it's not necessary and will be removed elsewhere
> > > in a follow-up patch.
> > > 
> > > With this in place, the lock need not be taken in execute_handler()
> > > any more. The uxlsnr only ever calls trylock() on the vecs lock,
> > > avoiding any waiting for other threads to finish.
> > > 
> > > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > > ---
> > >  multipathd/uxlsnr.c | 200 ++++++++++++++++++++++++++++------------
> > > ----
> > >  1 file changed, 129 insertions(+), 71 deletions(-)
> > > 
> > > diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
> > > index 87134d5..bf9780d 100644
> > > --- a/multipathd/uxlsnr.c
> > > +++ b/multipathd/uxlsnr.c
> > 
> > Nitpick: This would look clearer to me if, instead of a switch
> > statement, it was just
> > 
> > if (c->state != CLT_RECV)
> >         continue;
> > 
> > polls[i].events = POLLIN;
> > polls[i].fd = c->fd;
> > ...
> 
> That's true if you look at this patch in isolation. The reason I use a
> switch statement is that with patch 32, we get another case to treat
> here (CLT_SEND). At that point, the switch is at least on par wrt
> clarity, IMO.
> 
> No?

Yep. I retrack my objection

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>

> 
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 29/48] multipathd: uxlsnr: merge uxsock_trigger() into state machine

[Martin Wilck]

On Mon, 2021-11-29 at 10:19 -0600, Benjamin Marzinski wrote:
> On Fri, Nov 26, 2021 at 03:34:59PM +0100, Martin Wilck wrote:
> > 
> > I'm going to fix that, but I assume you're aware that our code is
> > far
> > from being consistent in this respect. This holds also for other
> > patches in this series. Do you want me to re-format all of them?
> 
> Huh? I mean that the line doesn't start with a tab, but instead has 8
> spaces.  A quick grep through the source code in your queue branch
> only
> turns that up in some of the files in the tests directory and in
> files
> we've just imported
> 
> # grep -l "^        " `find ./ -name "*.[ch]"`
> ./libmultipath/nvme/nvme-ioctl.c
> ./tests/pgpolicy.c
> ./tests/util.c
> ./tests/directio.c
> ./tests/mpathvalid.c
> ./tests/dmevents.c
> ./third-party/valgrind/drd.h
> ./third-party/valgrind/valgrind.h

Right. It must be some change in the way emacs handles indentation,
then. I'm not aware that I'm doing anything different than before.
Unfortunately, there are still some space-indented lines in my v3
submission.

I'll prepare a v4, and try to figure out what changed in my
environment. Sorry.

Martin



--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 32/48] multipathd: uxlsnr: use poll loop for sending, too

[Martin Wilck]

Hello Ben,

On Wed, 2021-11-24 at 19:43 -0600, Benjamin Marzinski wrote:
> On Thu, Nov 18, 2021 at 11:58:24PM +0100, mwilck@suse.com wrote:
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > send_packet() may busy-loop. By polling for POLLOUT, we can
> > avoid that, even if it's very unlikely in practice.
> > 
> 
> The last time I reviewed this, I mentioned that when we fall through
> from CLT_WORK to CLT_SEND, the client hasn't checked for a POLLOUT
> revent, so it's possible to block here. I'm not sure that we care.
> If not
> 
> Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>

Unlike the other patches, I did remove your "Reviewed-by:" on this
patch in the v3 series, because I made a non-trivial change in order
to fix the issue you mentioned. Now it's the only patch in the v3
series that doesn't have your Reviewed-by: tag.

Will you review this patch again? 

If you're fine with the patch, and you have no other objections,
I'll push the v4 of the series (with the whitespace issues fixed)
to my "queue2 branch and start preparing a submission for Christophe.

Regards
Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Benjamin Marzinski]

On Wed, Nov 24, 2021 at 10:20:53PM +0100, Martin Wilck wrote:
> On Wed, 2021-11-24 at 14:41 -0600, Benjamin Marzinski wrote:
> > On Thu, Nov 18, 2021 at 11:57:55PM +0100, mwilck@suse.com wrote:
> > > From: Martin Wilck <mwilck@suse.com>
> > > 
> > > Have struct mutex_lock take an optional wakeup function.
> > > unlock() is renamed to __unlock() in order to prevent it from
> > > being called by mistake.
> > > 
> > > This changes offsets in "struct vectors", requiring a major
> > > libmultipath version bump. While the strucure is already changed,
> > > in order to avoid this in the future, move the lock to the end
> > > of "struct vectors".
> > > 
> > > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > 
> > In libmultipath.version, I would have said that set_wakeup_fn was
> > added
> > in 10.0.0, instead of 9.2.0, which is a version that never actually
> > existed, but I don't think that's going to cause any problems so,
> > 
> > Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> 
> We should consider whether we want bump the version numbers only when
> we submit to Christophe, or maybe once per major patch series that
> touches the ABI. This is something that I could 

I suppose that we could just bump the version the first time after a
push to Christophe that an ABI change is made, and then just make all
the symbol changes under that version until Christophe makes another
release. If the first change only caused a minor version bump, and a
later change needed a major bump we could have two version bumps in a
release, but I don't think we're getting any benefit from making more.
But once per patchset works fine for me too.

-Ben
> 
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

On Mon, 2021-11-29 at 13:27 -0600, Benjamin Marzinski wrote:
> On Wed, Nov 24, 2021 at 10:20:53PM +0100, Martin Wilck wrote:
> > On Wed, 2021-11-24 at 14:41 -0600, Benjamin Marzinski wrote:
> > > On Thu, Nov 18, 2021 at 11:57:55PM +0100, mwilck@suse.com wrote:
> > > > From: Martin Wilck <mwilck@suse.com>
> > > > 
> > > > Have struct mutex_lock take an optional wakeup function.
> > > > unlock() is renamed to __unlock() in order to prevent it from
> > > > being called by mistake.
> > > > 
> > > > This changes offsets in "struct vectors", requiring a major
> > > > libmultipath version bump. While the strucure is already
> > > > changed,
> > > > in order to avoid this in the future, move the lock to the end
> > > > of "struct vectors".
> > > > 
> > > > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > > 
> > > In libmultipath.version, I would have said that set_wakeup_fn was
> > > added
> > > in 10.0.0, instead of 9.2.0, which is a version that never
> > > actually
> > > existed, but I don't think that's going to cause any problems so,
> > > 
> > > Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> > 
> > We should consider whether we want bump the version numbers only
> > when
> > we submit to Christophe, or maybe once per major patch series that
> > touches the ABI. This is something that I could 
> 
> I suppose that we could just bump the version the first time after a
> push to Christophe that an ABI change is made, and then just make all
> the symbol changes under that version until Christophe makes another
> release. If the first change only caused a minor version bump, and a
> later change needed a major bump we could have two version bumps in a
> release, but I don't think we're getting any benefit from making
> more.
> But once per patchset works fine for me too.

I agree. Also, I realize that we've bumped the library version too
often in the past. If we add a function, we don't need to bump. Because
a program that needs the added function will require e.g.
foo@LIBMULTIPATH_10.0.0, and this will fail for a library that doesn't
export foo (which is what we want). Likewise for function deletion - a
program that calls the deleted function will fail to link with the
updated library. OTOH, programs that use this version of the ABI
*without* using the functions which are added or removed are unaffected
by the addition / removal.

The only case in which the ABI version must be bumped is when we have
changed functions or data structures.

Furthermore, I believe now that the habit (which I introduced) to list
added functions at the end of the .version files, with comments
indicating when they were added, is useless. We should rather keep the
.version file ordered alphabetically.

Regards
Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Benjamin Marzinski]

On Mon, Nov 29, 2021 at 09:52:49PM +0100, Martin Wilck wrote:
> On Mon, 2021-11-29 at 13:27 -0600, Benjamin Marzinski wrote:
> > On Wed, Nov 24, 2021 at 10:20:53PM +0100, Martin Wilck wrote:
> > > On Wed, 2021-11-24 at 14:41 -0600, Benjamin Marzinski wrote:
> > > > On Thu, Nov 18, 2021 at 11:57:55PM +0100, mwilck@suse.com wrote:
> > > > > From: Martin Wilck <mwilck@suse.com>
> > > > > 
> > > > > Have struct mutex_lock take an optional wakeup function.
> > > > > unlock() is renamed to __unlock() in order to prevent it from
> > > > > being called by mistake.
> > > > > 
> > > > > This changes offsets in "struct vectors", requiring a major
> > > > > libmultipath version bump. While the strucure is already
> > > > > changed,
> > > > > in order to avoid this in the future, move the lock to the end
> > > > > of "struct vectors".
> > > > > 
> > > > > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > > > 
> > > > In libmultipath.version, I would have said that set_wakeup_fn was
> > > > added
> > > > in 10.0.0, instead of 9.2.0, which is a version that never
> > > > actually
> > > > existed, but I don't think that's going to cause any problems so,
> > > > 
> > > > Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>
> > > 
> > > We should consider whether we want bump the version numbers only
> > > when
> > > we submit to Christophe, or maybe once per major patch series that
> > > touches the ABI. This is something that I could 
> > 
> > I suppose that we could just bump the version the first time after a
> > push to Christophe that an ABI change is made, and then just make all
> > the symbol changes under that version until Christophe makes another
> > release. If the first change only caused a minor version bump, and a
> > later change needed a major bump we could have two version bumps in a
> > release, but I don't think we're getting any benefit from making
> > more.
> > But once per patchset works fine for me too.
> 
> I agree. Also, I realize that we've bumped the library version too
> often in the past. If we add a function, we don't need to bump. Because
> a program that needs the added function will require e.g.
> foo@LIBMULTIPATH_10.0.0, and this will fail for a library that doesn't
> export foo (which is what we want). Likewise for function deletion - a
> program that calls the deleted function will fail to link with the
> updated library. OTOH, programs that use this version of the ABI
> *without* using the functions which are added or removed are unaffected
> by the addition / removal.
> 
> The only case in which the ABI version must be bumped is when we have
> changed functions or data structures.
> 
> Furthermore, I believe now that the habit (which I introduced) to list
> added functions at the end of the .version files, with comments
> indicating when they were added, is useless. We should rather keep the
> .version file ordered alphabetically.

So we not use the minor version anymore? 

-Ben

> 
> Regards
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

On Tue, 2021-11-30 at 10:52 -0600, Benjamin Marzinski wrote:
> On Mon, Nov 29, 2021 at 09:52:49PM +0100, Martin Wilck wrote:
> > I agree. Also, I realize that we've bumped the library version too
> > often in the past. If we add a function, we don't need to bump.
> > Because
> > a program that needs the added function will require e.g.
> > foo@LIBMULTIPATH_10.0.0, and this will fail for a library that
> > doesn't
> > export foo (which is what we want). Likewise for function deletion
> > - a
> > program that calls the deleted function will fail to link with the
> > updated library. OTOH, programs that use this version of the ABI
> > *without* using the functions which are added or removed are
> > unaffected
> > by the addition / removal.
> > 
> > The only case in which the ABI version must be bumped is when we
> > have
> > changed functions or data structures.
> > 
> > Furthermore, I believe now that the habit (which I introduced) to
> > list
> > added functions at the end of the .version files, with comments
> > indicating when they were added, is useless. We should rather keep
> > the
> > .version file ordered alphabetically.
> 
> So we not use the minor version anymore? 

Perhaps we'll encounter another use case for it, or we find a flaw in
my reasoning above. I wouldn't remove the digit.

We could e.g. bump the minor version in an official release if there
are only minor ABI changes wrt the previous official release.

Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

On Tue, 2021-11-30 at 21:28 +0100, Martin Wilck wrote:
> On Tue, 2021-11-30 at 10:52 -0600, Benjamin Marzinski wrote:
> > On Mon, Nov 29, 2021 at 09:52:49PM +0100, Martin Wilck wrote:
> > > I agree. Also, I realize that we've bumped the library version
> > > too
> > > often in the past. If we add a function, we don't need to bump.
> > > Because
> > > a program that needs the added function will require e.g.
> > > foo@LIBMULTIPATH_10.0.0, and this will fail for a library that
> > > doesn't
> > > export foo (which is what we want). Likewise for function
> > > deletion
> > > - a
> > > program that calls the deleted function will fail to link with
> > > the
> > > updated library. OTOH, programs that use this version of the ABI
> > > *without* using the functions which are added or removed are
> > > unaffected
> > > by the addition / removal.
> > > 
> > > The only case in which the ABI version must be bumped is when we
> > > have
> > > changed functions or data structures.
> > > 
> > > Furthermore, I believe now that the habit (which I introduced) to
> > > list
> > > added functions at the end of the .version files, with comments
> > > indicating when they were added, is useless. We should rather
> > > keep
> > > the
> > > .version file ordered alphabetically.
> > 
> > So we not use the minor version anymore? 
> 
> Perhaps we'll encounter another use case for it, or we find a flaw in
> my reasoning above. I wouldn't remove the digit.

And here's the flaw: While my argument above is valid for ld.so, it's
not for package management tools like rpm. Here on openSUSE, we got rpm
Requires like "libmultipath.so.0(LIBMULTIPATH_13.0.0)(64bit)". As
distributors, we prefer incompatibilities to be detected at
installation time rather than at runtime. So, we do need the minor
version bumps.

Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Benjamin Marzinski]

On Wed, Dec 01, 2021 at 01:06:45PM +0100, Martin Wilck wrote:
> On Tue, 2021-11-30 at 21:28 +0100, Martin Wilck wrote:
> > On Tue, 2021-11-30 at 10:52 -0600, Benjamin Marzinski wrote:
> > > On Mon, Nov 29, 2021 at 09:52:49PM +0100, Martin Wilck wrote:
> > > > I agree. Also, I realize that we've bumped the library version
> > > > too
> > > > often in the past. If we add a function, we don't need to bump.
> > > > Because
> > > > a program that needs the added function will require e.g.
> > > > foo@LIBMULTIPATH_10.0.0, and this will fail for a library that
> > > > doesn't
> > > > export foo (which is what we want). Likewise for function
> > > > deletion
> > > > - a
> > > > program that calls the deleted function will fail to link with
> > > > the
> > > > updated library. OTOH, programs that use this version of the ABI
> > > > *without* using the functions which are added or removed are
> > > > unaffected
> > > > by the addition / removal.
> > > > 
> > > > The only case in which the ABI version must be bumped is when we
> > > > have
> > > > changed functions or data structures.
> > > > 
> > > > Furthermore, I believe now that the habit (which I introduced) to
> > > > list
> > > > added functions at the end of the .version files, with comments
> > > > indicating when they were added, is useless. We should rather
> > > > keep
> > > > the
> > > > .version file ordered alphabetically.
> > > 
> > > So we not use the minor version anymore? 
> > 
> > Perhaps we'll encounter another use case for it, or we find a flaw in
> > my reasoning above. I wouldn't remove the digit.
> 
> And here's the flaw: While my argument above is valid for ld.so, it's
> not for package management tools like rpm. Here on openSUSE, we got rpm
> Requires like "libmultipath.so.0(LIBMULTIPATH_13.0.0)(64bit)". As
> distributors, we prefer incompatibilities to be detected at
> installation time rather than at runtime. So, we do need the minor
> version bumps.

So where does this leave us. Are we bumping versions once per merge to
Christophe's offical branch, and if distributions put out multiple
releases between these, they are responsible for any version bumps that
they need.  Or are we bumping versions once per patchset if needed?

-Ben

> 
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

On Tue, 2021-12-14 at 17:25 -0600, Benjamin Marzinski wrote:
> On Wed, Dec 01, 2021 at 01:06:45PM +0100, Martin Wilck wrote:
> > On Tue, 2021-11-30 at 21:28 +0100, Martin Wilck wrote:
> > > 
> > > Perhaps we'll encounter another use case for it, or we find a
> > > flaw in
> > > my reasoning above. I wouldn't remove the digit.
> > 
> > And here's the flaw: While my argument above is valid for ld.so,
> > it's
> > not for package management tools like rpm. Here on openSUSE, we got
> > rpm
> > Requires like "libmultipath.so.0(LIBMULTIPATH_13.0.0)(64bit)". As
> > distributors, we prefer incompatibilities to be detected at
> > installation time rather than at runtime. So, we do need the minor
> > version bumps.
> 
> So where does this leave us. Are we bumping versions once per merge
> to
> Christophe's offical branch, and if distributions put out multiple
> releases between these, they are responsible for any version bumps
> that
> they need.  Or are we bumping versions once per patchset if needed?

I just wanted to say that my previous argument that addition or removal
of symbols can be ignored was wrong, and that we should keep the
treatment of minor versions as originally designed. I am fine with us
not touching the .version files while we work on new patch sets, and
leaving it to the person perparing a PR for Christope (likely myself)
to fix it up when a patch series is finished. In practice, it'll
probably result in just a major version bump per submission to
Christophe, but that isn't cast in stone (if we submit a smaller set of
patches, it might be just a minor bump, or none at all).

Distros are free to modify the last digit as they please.

Martin


--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Martin Wilck]

Hi Ben,

On Wed, 2021-12-15 at 08:33 +0100, Martin Wilck wrote:
> 
> I just wanted to say that my previous argument that addition or
> removal
> of symbols can be ignored was wrong, and that we should keep the
> treatment of minor versions as originally designed. I am fine with us
> not touching the .version files while we work on new patch sets, and
> leaving it to the person perparing a PR for Christope (likely myself)
> to fix it up when a patch series is finished. In practice, it'll
> probably result in just a major version bump per submission to
> Christophe, but that isn't cast in stone (if we submit a smaller set
> of
> patches, it might be just a minor bump, or none at all).
> 
> Distros are free to modify the last digit as they please.

Here's what I think I should do: I'll keep one patch on top of the
"queue" branch that includes the necessary ABI bumps. I'll fix this
patch up as commits are added to "queue". This means the queue branch
(more precisely, the topmost commit) will need to be rebased. It won't
be a problem as this commit will only touch the .version files, nothing
else. This way builds from "queue" will be "safe" against library
incompatibilities wrt the official release, and yet we won't need to
bump the major version multiple times for a single submission to
Christophe. Submissions to Christophe will come with either no bump, or
a single-step minor bump, or a single-step major bump.

Does this make sense?

Regards
Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v2 03/48] libmultipath: add optional wakeup functionality to lock.c

[Benjamin Marzinski]

On Wed, Dec 15, 2021 at 10:01:34PM +0100, Martin Wilck wrote:
> Hi Ben,
> 
> On Wed, 2021-12-15 at 08:33 +0100, Martin Wilck wrote:
> > 
> > I just wanted to say that my previous argument that addition or
> > removal
> > of symbols can be ignored was wrong, and that we should keep the
> > treatment of minor versions as originally designed. I am fine with us
> > not touching the .version files while we work on new patch sets, and
> > leaving it to the person perparing a PR for Christope (likely myself)
> > to fix it up when a patch series is finished. In practice, it'll
> > probably result in just a major version bump per submission to
> > Christophe, but that isn't cast in stone (if we submit a smaller set
> > of
> > patches, it might be just a minor bump, or none at all).
> > 
> > Distros are free to modify the last digit as they please.
> 
> Here's what I think I should do: I'll keep one patch on top of the
> "queue" branch that includes the necessary ABI bumps. I'll fix this
> patch up as commits are added to "queue". This means the queue branch
> (more precisely, the topmost commit) will need to be rebased. It won't
> be a problem as this commit will only touch the .version files, nothing
> else. This way builds from "queue" will be "safe" against library
> incompatibilities wrt the official release, and yet we won't need to
> bump the major version multiple times for a single submission to
> Christophe. Submissions to Christophe will come with either no bump, or
> a single-step minor bump, or a single-step major bump.
> 
> Does this make sense?
> 
Sure. I'm fine with that.

-Ben

> Regards
> Martin

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel