[PATCH 00/57] SLES resync

[PATCH 00/57] SLES resync

[Hannes Reinecke]

Hi all,

as promised several times, here's now a patchset with all the patches
queued up in my SLES repository.

The first bits are pretty much uncontroversial (I hope), mainly fixes
etc. Next there are some systemd updates, setting dependencies and such.

The _really_ important patch is

multipathd: skip uninitialized devices during reconfiguration

which basically is the cause for all the missing wwid issues we've seen.
So I wonder if we shouldn't just do away with all the tweaks we've
accumlated so far and see if that patch isn't sufficient.
According to Kay Sievers it should...

Next is a general overhaul of the 'cookie' mechanism; Ben, Mike,
mind to have a look here? It _seems_ reasonable enough, but it would
be nice to have additional confirmation for that.

Finally there are patches which probably require some more discussion:
- Asynchronous configuration: Do not call 'reconfigure' directly, but
  rather set the daemon_status to 'configuration' and defer the actual
  work to the main thread. This allows us to decouple it from the CLI
  call, and we won't run into any udev/systemd timeout issues when
  booting on large installations
- Lock pushdown: I remember to have sent this patch already at one
  point, but it got rejected. So I try again :-)
  With this patch the big vector lock is pushed down to the areas
  where we actually need it, making the entire system running far
  smoother during booting or failover in large instances.
  We have been using this patch since SLE12 GA, ie for over two
  years now. And seem to have ironed out all issues with it.

The entire patchset can be found at:

github.com:/hreinecke/multipath-tools.git
branch sles-resync

As usual, comments and reviews are welcome.
Patches are based upon the latest git, including Bens patchset.

Hannes Reinecke (57):
  kpartx: Fixup persistent name generation
  kpartx: handle more than 256 loop devices
  kpartx: parse emulated DASD devices
  kpartx: Install rules file with correct prefix
  Add HP MSA 2040 to the hardware table
  Use ALUA for HP 3PAR
  Add LIO-ORG/SUSE RBD backend hardware defaults
  Allow for empty SCSI revision
  libmultipath: Do not use 'sscanf' for parsing integers
  libmultipath: finally fix dev_loss_tmo setting
  multipathd: fixup queueing mode in 'show maps status'
  multipathd: fixup a crash when invoking CLI commands
  multipathd: print error message for invalid arguments
  libmultipath: correctly initialize pp->sg_id
  libmultipath: correctly display checker status
  libmultipath: call get_uid() for all paths
  libmultipath: additional logging messages when formatting callout
  libmpathpersist: Fixup whitespaces in Makefile
  multipathd: Do not print misleading message 'not found in pathvec'
  multipathd: Do not update the paths vec when removing paths
  libmultipath: avoid double semicolon in lock.h
  multipath: use option '-i' when called from udev
  multipath: remove warning 'failed to get wwid'
  Add dependency on systemd-udevd.service
  Load all device handler modules on startup
  11-dm-mpath.rules: Only import ID_FS_XXX variables if not set
  Ensure multipathd is started before systemd-udev-trigger.service
  libmultipath: use a shared lock to co-operate with udev
  Only filter for udev property if uid_attribute is present
  multipathd: skip uninitialized devices during reconfiguration
  multipathd: improve uxlsnr
  libmultipath: remove 'needsync' argument from dm_simplecmd_noflush
  libmultipath: remove 'use_uuid' argument from dm_addmap()
  Always set DM_UDEV_DISABLE_LIBRARY_FALLBACK
  libmultipath: pass in cookie as argument for dm_simplecmd()
  libmultipath: pass in 'cookie' as argument for dm_addmap()
  Remove 'udev_sync' argument from dm_simplecmd()
  libmultipath: Fixup 'DM_DEVICE_RELOAD' handling
  devmapper: do not flush I/O for DM_DEVICE_CREATE
  libmultipath: fixup dm_rename to complete cookie on failure
  multipathd: accept zero-size paths in ev_add_path()
  Use multipath wwid if path wwid is empty
  multipathd: set uxsock_timeout after reconfiguration
  multipathd: Do not switch paths on empty multipath tables
  libmultipath: remove 'get_info' argument for adopt_paths()
  libmultipath: ensure 'dev_t' is set when store paths
  libmultipath: sanity check on store_path()
  dmparser: Use find_path_by_dev()
  multipathd: do not flush maps on startup
  multipathd: strict loop timings
  multipathd: Provide standard error description on cli failure
  libmultipath: make 'dm_addmap' static
  multipathd: implement 'show map $map format $fmt'
  multipathd: Increase uxclnt timeout
  multipathd: asynchronous configuration
  multipathd: push down lock in checkerloop()
  Allow specific CLI commands to run unlocked

 kpartx/Makefile               |   6 +-
 kpartx/dasd.c                 |  39 +++-
 kpartx/dasd.h                 |   1 +
 kpartx/kpartx.rules           |  11 +-
 kpartx/kpartx_id              |  11 +-
 kpartx/lopart.c               | 106 +++++----
 libmpathpersist/Makefile      |   4 +-
 libmultipath/callout.c        |   6 +-
 libmultipath/config.h         |   1 +
 libmultipath/configure.c      |  33 ++-
 libmultipath/defaults.h       |   1 +
 libmultipath/devmapper.c      | 114 ++++++----
 libmultipath/devmapper.h      |   4 +-
 libmultipath/dict.c           |   4 +
 libmultipath/discovery.c      | 115 +++++-----
 libmultipath/dmparser.c       |  20 +-
 libmultipath/hwtable.c        |  41 +++-
 libmultipath/lock.h           |   4 +-
 libmultipath/print.c          |   6 +-
 libmultipath/structs.c        |  17 +-
 libmultipath/structs_vec.c    |  10 +-
 libmultipath/structs_vec.h    |   2 +-
 libmultipath/uevent.c         |  10 +-
 multipath/11-dm-mpath.rules   |  10 +-
 multipath/main.c              |   1 -
 multipath/multipath.conf.5    |  16 +-
 multipath/multipath.rules     |   2 +-
 multipathd/cli.c              | 101 +++++++--
 multipathd/cli.h              |   4 +-
 multipathd/cli_handlers.c     |  85 +++++++-
 multipathd/cli_handlers.h     |   2 +
 multipathd/main.c             | 486 +++++++++++++++++++++++++++++-------------
 multipathd/main.h             |   3 +
 multipathd/multipathd.service |   6 +-
 multipathd/uxlsnr.c           |  97 +++++----
 35 files changed, 923 insertions(+), 456 deletions(-)

-- 
2.6.6

[PATCH 01/57] kpartx: Fixup persistent name generation

[Hannes Reinecke]

kpartx_id would overwrite the 'DM_NAME' variable, which were
causing things to break later on. Seeing that kpartx_id is
only ever used to generate persistent device names I've updated
kpartx_id and kpartx.rules to rely on existing variables where
possible, avoiding any accidental overwrite of existing variables.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 kpartx/kpartx.rules | 11 +++--------
 kpartx/kpartx_id    | 11 +----------
 2 files changed, 4 insertions(+), 18 deletions(-)

diff --git a/kpartx/kpartx.rules b/kpartx/kpartx.rules
index 022361f..f6a3706 100644
--- a/kpartx/kpartx.rules
+++ b/kpartx/kpartx.rules
@@ -14,19 +14,14 @@ ENV{DM_UUID}=="?*", IMPORT{program}=="kpartx_id %M %m $env{DM_UUID}"
 
 OPTIONS="link_priority=50"
 
-# Create persistent links for multipath tables
-ENV{DM_UUID}=="mpath-*", \
+ENV{DM_UUID}=="?*", ENV{DM_TYPE}=="?*" \
 	SYMLINK+="disk/by-id/$env{DM_TYPE}-$env{DM_NAME}"
-ENV{DM_MPATH}=="?*", ENV{DM_PART}!="?*", \
-	SYMLINK+="disk/by-id/$env{DM_TYPE}-$env{DM_MPATH}"
+
+# Create persistent links for multipath tables
 ENV{DM_WWN}=="?*", ENV{DM_PART}!="?*", \
 	SYMLINK+="disk/by-id/wwn-$env{DM_WWN}"
 
 # Create persistent links for partitions
-ENV{DM_PART}=="?*", \
-        SYMLINK+="disk/by-id/$env{DM_TYPE}-$env{DM_NAME}-part$env{DM_PART}"
-ENV{DM_MPATH}=="?*", ENV{DM_PART}=="?*", \
-	SYMLINK+="disk/by-id/$env{DM_TYPE}-$env{DM_MPATH}-part$env{DM_PART}"
 ENV{DM_WWN}=="?*", ENV{DM_PART}=="?*", \
 	SYMLINK+="disk/by-id/wwn-$env{DM_WWN}-part$env{DM_PART}"
 
diff --git a/kpartx/kpartx_id b/kpartx/kpartx_id
index 517b856..df0e6b9 100644
--- a/kpartx/kpartx_id
+++ b/kpartx/kpartx_id
@@ -52,12 +52,8 @@ fi
 # Set the name of the table. We're only interested in dmraid,
 # multipath, and kpartx tables; everything else is ignored.
 if [ "$dmtbl" = "part" ] ; then
-    # The name of the kpartx table is the name of the parent table
     dmname=$($DMSETUP info  -c --noheadings -o name -u $dmuuid)
-    echo "DM_NAME=$dmname"
-    if [ "$dmname" != ${dmuuid#mpath-} ] ; then
-	echo "DM_MPATH=${dmuuid#mpath-}"
-    fi
+    echo "DM_MPATH=$dmname"
     # We need the dependencies of the parent table to figure out
     # the type if the parent is a multipath table
     case "$dmuuid" in
@@ -66,14 +62,9 @@ if [ "$dmtbl" = "part" ] ; then
 	    ;;
     esac
 elif [ "$dmtbl" = "mpath" ] ; then
-    if [ -n "$DM_NAME" -a "$DM_NAME" != "$dmuuid" ] ; then
-	echo "DM_MPATH=$dmuuid"
-    fi
     dmname="$dmuuid"
     # We need the dependencies of the table to figure out the type
     dmdeps=$($DMSETUP deps -u $UUID)
-elif [ "$dmtbl" = "dmraid" ] ; then
-    dmname=$tblname
 fi
 
 [ -n "$dmpart" ] && echo "DM_PART=$dmpart"
-- 
2.6.6

[PATCH 02/57] kpartx: handle more than 256 loop devices

[Hannes Reinecke]

Update the loop handling code to support more than 256
devices.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 kpartx/lopart.c | 106 ++++++++++++++++++++++++++------------------------------
 1 file changed, 49 insertions(+), 57 deletions(-)

diff --git a/kpartx/lopart.c b/kpartx/lopart.c
index 26f3011..ffde6d9 100644
--- a/kpartx/lopart.c
+++ b/kpartx/lopart.c
@@ -25,8 +25,9 @@
 #include <sys/ioctl.h>
 #include <sys/stat.h>
 #include <sys/mman.h>
+#include <sys/types.h>
+#include <dirent.h>
 #include <sysmacros.h>
-#include <asm/posix_types.h>
 #include <linux/loop.h>
 
 #include "lopart.h"
@@ -98,78 +99,71 @@ is_loop_device (const char *device)
 extern char *
 find_loop_by_file (const char * filename)
 {
-	char dev[64];
-	char *loop_formats[] = { "/dev/loop%d", "/dev/loop/%d" };
-	int i, j, fd;
+	DIR *dir;
+	struct dirent *dent;
+	char dev[64], *found = NULL;
+	int fd;
 	struct stat statbuf;
 	struct loop_info loopinfo;
-	dev_t file_dev;
-	ino_t file_ino;
 
-	if (stat (filename, &statbuf) != 0) {
+	dir = opendir("/dev");
+	if (!dir)
 		return NULL;
-	}
-	file_dev = statbuf.st_dev;
-	file_ino = statbuf.st_ino;
-
-	for (j = 0; j < SIZE(loop_formats); j++) {
-
-		for (i = 0; i < 256; i++) {
-			sprintf (dev, loop_formats[j], i);
-
-			if (stat (dev, &statbuf) != 0 ||
-			    !S_ISBLK(statbuf.st_mode))
-				continue;
-
-			fd = open (dev, O_RDONLY);
 
-			if (fd < 0)
-				break;
+	while ((dent = readdir(dir)) != NULL) {
+		if (strncmp(dent->d_name,"loop",4))
+			continue;
+		if (!strcmp(dent->d_name, "loop-control"))
+			continue;
+		sprintf(dev, "/dev/%s", dent->d_name);
 
-			if (ioctl (fd, LOOP_GET_STATUS, &loopinfo) != 0) {
-				close (fd);
-				continue;
-			}
+		if (stat (dev, &statbuf) != 0 ||
+		    !S_ISBLK(statbuf.st_mode))
+			continue;
 
-			if (loopinfo.lo_device == file_dev && loopinfo.lo_inode == file_ino) {
-				close (fd);
-				return xstrdup(dev); /*found */
-			}
+		fd = open (dev, O_RDONLY);
+		if (fd < 0)
+			break;
 
+		if (ioctl (fd, LOOP_GET_STATUS, &loopinfo) != 0) {
 			close (fd);
 			continue;
 		}
+
+		if (0 == strcmp(filename, loopinfo.lo_name)) {
+			close (fd);
+			found = xstrdup(dev);
+			break;
+		}
+
+		close (fd);
 	}
-	return NULL;
+	closedir(dir);
+	return found;
 }
 
 extern char *
 find_unused_loop_device (void)
 {
-	/* Just creating a device, say in /tmp, is probably a bad idea -
-	   people might have problems with backup or so.
-	   So, we just try /dev/loop[0-7]. */
-
-	char dev[20];
-	char *loop_formats[] = { "/dev/loop%d", "/dev/loop/%d" };
-	int i, j, fd, first = 0, somedev = 0, someloop = 0, loop_known = 0;
+	char dev[20], *next_loop_dev = NULL;;
+	int fd, next_loop = 0, somedev = 0, someloop = 0, loop_known = 0;
 	struct stat statbuf;
 	struct loop_info loopinfo;
 	FILE *procdev;
 
-	if (stat("/dev/loop-control", &statbuf) == 0 &&
-	    S_ISCHR(statbuf.st_mode)) {
-		fd = open("/dev/loop-control", O_RDWR);
-		if (fd >= 0)
-			first = ioctl(fd, LOOP_CTL_GET_FREE);
-		close(fd);
-		if (first < 0)
-			first = 0;
-	}
-	for (j = 0; j < SIZE(loop_formats); j++) {
+	while (next_loop_dev == NULL) {
+		if (stat("/dev/loop-control", &statbuf) == 0 &&
+		    S_ISCHR(statbuf.st_mode)) {
+			fd = open("/dev/loop-control", O_RDWR);
+			if (fd < 0)
+				return NULL;
+			next_loop = ioctl(fd, LOOP_CTL_GET_FREE);
+			if (next_loop < 0)
+				return NULL;
+			close(fd);
+		}
 
-	    for(i = first; i < 256; i++) {
-		sprintf(dev, loop_formats[j], i);
+		sprintf(dev, "/dev/loop%d", next_loop);
 
 		if (stat (dev, &statbuf) == 0 && S_ISBLK(statbuf.st_mode)) {
 			somedev++;
@@ -182,18 +176,19 @@ find_unused_loop_device (void)
 
 				else if (errno == ENXIO) {
 					close (fd);
-					return xstrdup(dev);/* probably free */
+					next_loop_dev = xstrdup(dev);
 				}
 
 				close (fd);
 			}
-			
+
 			/* continue trying as long as devices exist */
 			continue;
 		}
 		break;
-	    }
 	}
+	if (next_loop_dev)
+		return next_loop_dev;
 
 	/* Nothing found. Why not? */
 	if ((procdev = fopen(PROC_DEVICES, "r")) != NULL) {
@@ -216,12 +211,10 @@ find_unused_loop_device (void)
 		fprintf(stderr, "mount: could not find any device /dev/loop#");
 
 	else if (!someloop) {
-
 	    if (loop_known == 1)
 		fprintf(stderr,
 		    "mount: Could not find any loop device.\n"
 		    "       Maybe /dev/loop# has a wrong major number?");
-	    
 	    else if (loop_known == -1)
 		fprintf(stderr,
 		    "mount: Could not find any loop device, and, according to %s,\n"
@@ -237,7 +230,6 @@ find_unused_loop_device (void)
 
 	} else
 		fprintf(stderr, "mount: could not find any free loop device");
-	
 	return 0;
 }
 
-- 
2.6.6

[PATCH 03/57] kpartx: parse emulated DASD devices

[Hannes Reinecke]

Emulated DASD devices do not support the DASDINFO ioctl, so we
need to generate the information by hand. After all, fdasd
does the same.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 kpartx/dasd.c | 39 +++++++++++++++++++++++++++++++++------
 kpartx/dasd.h |  1 +
 2 files changed, 34 insertions(+), 6 deletions(-)

diff --git a/kpartx/dasd.c b/kpartx/dasd.c
index 1fcf778..14c35b1 100644
--- a/kpartx/dasd.c
+++ b/kpartx/dasd.c
@@ -46,6 +46,21 @@ unsigned long long sectors512(unsigned long long sectors, int blocksize)
 	return sectors * (blocksize >> 9);
 }
 
+/*
+ * Magic records per track calculation, copied from fdasd.c
+ */
+static unsigned int ceil_quot(unsigned int d1, unsigned int d2)
+{
+	return (d1 + (d2 - 1)) / d2;
+}
+
+unsigned int recs_per_track(unsigned int dl)
+{
+	int dn = ceil_quot(dl + 6, 232) + 1;
+	return 1729 / (10 + 9 + ceil_quot(dl + 6 * dn, 34));
+}
+
+
 typedef unsigned int __attribute__((__may_alias__)) label_ints_t;
 
 /*
@@ -124,19 +139,31 @@ read_dasd_pt(int fd, struct slice all, struct slice *sp, int ns)
 	}
 
 	if (ioctl(fd_dasd, BIODASDINFO, (unsigned long)&info) != 0) {
-		goto out;
+		info.label_block = 2;
+		info.FBA_layout = 0;
+		memcpy(info.type, "ECKD", sizeof(info.type));
 	}
 
-	if (ioctl(fd_dasd, HDIO_GETGEO, (unsigned long)&geo) != 0) {
+	if (ioctl(fd_dasd, BLKSSZGET, &blocksize) != 0)
 		goto out;
-	}
 
 	if (ioctl(fd_dasd, BLKGETSIZE64, &disksize) != 0)
 		goto out;
-	disksize >>= 9;
 
-	if (ioctl(fd_dasd, BLKSSZGET, &blocksize) != 0)
-		goto out;
+	if (ioctl(fd_dasd, HDIO_GETGEO, (unsigned long)&geo) != 0) {
+		unsigned int cyl;
+
+		geo.heads = 15;
+		geo.sectors = recs_per_track(blocksize);
+		cyl = disksize / (blocksize * geo.heads * geo.sectors);
+		if (cyl < LV_COMPAT_CYL)
+			geo.cylinders = cyl;
+		else
+			geo.cylinders = LV_COMPAT_CYL;
+		geo.start = 0;
+	}
+
+	disksize >>= 9;
 
 	if (blocksize < 512 || blocksize > 4096)
 		goto out;
diff --git a/kpartx/dasd.h b/kpartx/dasd.h
index 42f94db..6fa64e6 100644
--- a/kpartx/dasd.h
+++ b/kpartx/dasd.h
@@ -164,6 +164,7 @@ typedef struct dasd_information_t {
 #define BLKGETSIZE _IO(0x12,96)
 #define BLKSSZGET _IO(0x12,104)
 #define BLKGETSIZE64 _IOR(0x12,114,size_t) /* device size in bytes (u64 *arg)*/
+#define LV_COMPAT_CYL 0xFFFE
 
 /*
  * Only compile this on S/390. Doesn't make any sense
-- 
2.6.6

[PATCH 04/57] kpartx: Install rules file with correct prefix

[Hannes Reinecke]

We don't need to swizzle names in the spec file, the Makefile
should be doing it on itself.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 kpartx/Makefile | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/kpartx/Makefile b/kpartx/Makefile
index 4ba38ba..501b277 100644
--- a/kpartx/Makefile
+++ b/kpartx/Makefile
@@ -28,8 +28,8 @@ install: $(EXEC) $(EXEC).8
 	$(INSTALL_PROGRAM) -m 755 $(EXEC) $(DESTDIR)$(bindir)
 	$(INSTALL_PROGRAM) -d $(DESTDIR)$(libudevdir)
 	$(INSTALL_PROGRAM) -m 755 kpartx_id $(DESTDIR)$(libudevdir)
-	$(INSTALL_PROGRAM) -d $(DESTDIR)/etc/udev/rules.d
-	$(INSTALL_PROGRAM) -m 644 kpartx.rules $(DESTDIR)/etc/udev/rules.d/
+	$(INSTALL_PROGRAM) -d $(DESTDIR)$(libudevdir)/rules.d
+	$(INSTALL_PROGRAM) -m 644 kpartx.rules $(DESTDIR)$(libudevdir)/rules.d/66-kpartx.rules
 	$(INSTALL_PROGRAM) -d $(DESTDIR)$(mandir)
 	$(INSTALL_PROGRAM) -m 644 $(EXEC).8.gz $(DESTDIR)$(mandir)
 
@@ -37,6 +37,8 @@ uninstall:
 	rm -f $(DESTDIR)$(bindir)/$(EXEC)
 	rm -f $(DESTDIR)$(mandir)/$(EXEC).8.gz
 	rm -f $(DESTDIR)$(libudevdir)/kpartx_id
+	rm -f $(DESTDIR)$(libudevdir)/rules.d/66-kpartx.rules
+	rm -f $(DESTDIR)$(libudevdir)/rules.d/67-kpartx-compat.rules
 
 clean:
 	rm -f core *.o $(EXEC) *.gz
-- 
2.6.6

[PATCH 05/57] Add HP MSA 2040 to the hardware table

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/hwtable.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
index a4ae053..28ee595 100644
--- a/libmultipath/hwtable.c
+++ b/libmultipath/hwtable.c
@@ -175,6 +175,21 @@ static struct hwentry default_hw[] = {
 		.prio_name     = PRIO_ALUA,
 		.prio_args     = NULL,
 	},
+	{
+		/* HP MSA 1040/2040 product family */
+		.vendor        = "HP",
+		.product       = "MSA (1|2)040 SA(N|S)",
+		.features      = DEFAULT_FEATURES,
+		.hwhandler     = DEFAULT_HWHANDLER,
+		.pgpolicy      = GROUP_BY_PRIO,
+		.pgfailback    = -FAILBACK_IMMEDIATE,
+		.rr_weight     = RR_WEIGHT_NONE,
+		.no_path_retry = 18,
+		.minio         = 100,
+		.checker_name  = TUR,
+		.prio_name     = PRIO_ALUA,
+		.prio_args     = NULL,
+	},
 
 	{
 		/* HP SVSP */
-- 
2.6.6

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Sebastian Herbszt]

Hannes Reinecke wrote:
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/hwtable.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)

You missed adding it to multipath.conf.defaults.

> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
> index a4ae053..28ee595 100644
> --- a/libmultipath/hwtable.c
> +++ b/libmultipath/hwtable.c
> @@ -175,6 +175,21 @@ static struct hwentry default_hw[] = {
>  		.prio_name     = PRIO_ALUA,
>  		.prio_args     = NULL,
>  	},
> +	{
> +		/* HP MSA 1040/2040 product family */
> +		.vendor        = "HP",
> +		.product       = "MSA (1|2)040 SA(N|S)",
> +		.features      = DEFAULT_FEATURES,
> +		.hwhandler     = DEFAULT_HWHANDLER,
> +		.pgpolicy      = GROUP_BY_PRIO,
> +		.pgfailback    = -FAILBACK_IMMEDIATE,
> +		.rr_weight     = RR_WEIGHT_NONE,
> +		.no_path_retry = 18,
> +		.minio         = 100,
> +		.checker_name  = TUR,
> +		.prio_name     = PRIO_ALUA,
> +		.prio_args     = NULL,
> +	},
>  
>  	{
>  		/* HP SVSP */

Any reason for a separate entry and not merging it with
"HP MSA2000 product family with new firmware" ?

Sebastian

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Hannes Reinecke]

On 04/29/2016 12:06 AM, Sebastian Herbszt wrote:
> Hannes Reinecke wrote:
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  libmultipath/hwtable.c | 15 +++++++++++++++
>>  1 file changed, 15 insertions(+)
> 
> You missed adding it to multipath.conf.defaults.
> 
Yes, but we're about to phase them out anyway.
Cf the recent discussion.

>> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
>> index a4ae053..28ee595 100644
>> --- a/libmultipath/hwtable.c
>> +++ b/libmultipath/hwtable.c
>> @@ -175,6 +175,21 @@ static struct hwentry default_hw[] = {
>>  		.prio_name     = PRIO_ALUA,
>>  		.prio_args     = NULL,
>>  	},
>> +	{
>> +		/* HP MSA 1040/2040 product family */
>> +		.vendor        = "HP",
>> +		.product       = "MSA (1|2)040 SA(N|S)",
>> +		.features      = DEFAULT_FEATURES,
>> +		.hwhandler     = DEFAULT_HWHANDLER,
>> +		.pgpolicy      = GROUP_BY_PRIO,
>> +		.pgfailback    = -FAILBACK_IMMEDIATE,
>> +		.rr_weight     = RR_WEIGHT_NONE,
>> +		.no_path_retry = 18,
>> +		.minio         = 100,
>> +		.checker_name  = TUR,
>> +		.prio_name     = PRIO_ALUA,
>> +		.prio_args     = NULL,
>> +	},
>>  
>>  	{
>>  		/* HP SVSP */
> 
> Any reason for a separate entry and not merging it with
> "HP MSA2000 product family with new firmware" ?
> 
Yes. MSA2000 are completely different beasts, so I'd like to keep
them separate.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Sebastian Herbszt]

Hannes Reinecke wrote:
> On 04/29/2016 12:06 AM, Sebastian Herbszt wrote:
> > Hannes Reinecke wrote:
> >> Signed-off-by: Hannes Reinecke <hare@suse.de>
> >> ---
> >>  libmultipath/hwtable.c | 15 +++++++++++++++
> >>  1 file changed, 15 insertions(+)
> > 
> > You missed adding it to multipath.conf.defaults.
> > 
> Yes, but we're about to phase them out anyway.
> Cf the recent discussion.

Looks like that part hid in the "multipath-0.5.0 still provides broken
udev rules" [1] thread. Thanks for the hint.

[1] https://www.redhat.com/archives/dm-devel/2016-April/msg00458.html

Sebastian

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Xose Vazquez Perez]

On 04/29/2016 07:55 AM, Hannes Reinecke wrote:

> On 04/29/2016 12:06 AM, Sebastian Herbszt wrote:
>>
>>> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
>>> index a4ae053..28ee595 100644
>>> --- a/libmultipath/hwtable.c
>>> +++ b/libmultipath/hwtable.c
>>> @@ -175,6 +175,21 @@ static struct hwentry default_hw[] = {
>>>  		.prio_name     = PRIO_ALUA,
>>>  		.prio_args     = NULL,
>>>  	},
>>> +	{
>>> +		/* HP MSA 1040/2040 product family */
>>> +		.vendor        = "HP",
>>> +		.product       = "MSA (1|2)040 SA(N|S)",
>>> +		.features      = DEFAULT_FEATURES,
>>> +		.hwhandler     = DEFAULT_HWHANDLER,
>>> +		.pgpolicy      = GROUP_BY_PRIO,
>>> +		.pgfailback    = -FAILBACK_IMMEDIATE,
>>> +		.rr_weight     = RR_WEIGHT_NONE,
>>> +		.no_path_retry = 18,
>>> +		.minio         = 100,
>>> +		.checker_name  = TUR,
>>> +		.prio_name     = PRIO_ALUA,
>>> +		.prio_args     = NULL,
>>> +	},
>>>  
>>>  	{
>>>  		/* HP SVSP */
>>
>> Any reason for a separate entry and not merging it with
>> "HP MSA2000 product family with new firmware" ?
>>
> Yes. MSA2000 are completely different beasts, so I'd like to keep
> them separate.

Sebastian is right.

And these three can be folded into one, because they share *exactly* the
same configuration.

/* HP MSA2000 product family with new firmware */
.vendor        = "HP",
.product       = "MSA2012sa|MSA23(12|24)(fc|i|sa)|MSA2000s VOLUME",

/* HP P2000 family arrays */
.vendor        = "HP",
.product       = "P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI",

/* HP MSA 1040/2040 product family */
.vendor        = "HP",
.product       = "MSA (1|2)040 SA(N|S)",


In VxVM(libvxmsa2kfc_sa) they are grouped that way:
https://web.archive.org/web/20160608233140/https://sort.veritas.com/asl/details/756

Otherwise if hwtable.c is filled with a lot of duplicate entries,
it could be unmanageable.

Also all three DELL, some IBM, .... could be regrouped.

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Hannes Reinecke]

On 06/09/2016 04:20 PM, Xose Vazquez Perez wrote:
> On 04/29/2016 07:55 AM, Hannes Reinecke wrote:
> 
>> On 04/29/2016 12:06 AM, Sebastian Herbszt wrote:
>>>
>>>> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
>>>> index a4ae053..28ee595 100644
>>>> --- a/libmultipath/hwtable.c
>>>> +++ b/libmultipath/hwtable.c
>>>> @@ -175,6 +175,21 @@ static struct hwentry default_hw[] = {
>>>>  		.prio_name     = PRIO_ALUA,
>>>>  		.prio_args     = NULL,
>>>>  	},
>>>> +	{
>>>> +		/* HP MSA 1040/2040 product family */
>>>> +		.vendor        = "HP",
>>>> +		.product       = "MSA (1|2)040 SA(N|S)",
>>>> +		.features      = DEFAULT_FEATURES,
>>>> +		.hwhandler     = DEFAULT_HWHANDLER,
>>>> +		.pgpolicy      = GROUP_BY_PRIO,
>>>> +		.pgfailback    = -FAILBACK_IMMEDIATE,
>>>> +		.rr_weight     = RR_WEIGHT_NONE,
>>>> +		.no_path_retry = 18,
>>>> +		.minio         = 100,
>>>> +		.checker_name  = TUR,
>>>> +		.prio_name     = PRIO_ALUA,
>>>> +		.prio_args     = NULL,
>>>> +	},
>>>>  
>>>>  	{
>>>>  		/* HP SVSP */
>>>
>>> Any reason for a separate entry and not merging it with
>>> "HP MSA2000 product family with new firmware" ?
>>>
>> Yes. MSA2000 are completely different beasts, so I'd like to keep
>> them separate.
> 
> Sebastian is right.
> 
> And these three can be folded into one, because they share *exactly* the
> same configuration.
> 
Note that I didn't say they _cannot_ be merged.
I've said that I would _like_ to keep them separate, being as they are
totally different hardware-wise.


> /* HP MSA2000 product family with new firmware */
> .vendor        = "HP",
> .product       = "MSA2012sa|MSA23(12|24)(fc|i|sa)|MSA2000s VOLUME",
> 
> /* HP P2000 family arrays */
> .vendor        = "HP",
> .product       = "P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI",
> 
> /* HP MSA 1040/2040 product family */
> .vendor        = "HP",
> .product       = "MSA (1|2)040 SA(N|S)",
> 
> 
> In VxVM(libvxmsa2kfc_sa) they are grouped that way:
> https://web.archive.org/web/20160608233140/https://sort.veritas.com/asl/details/756
> 
> Otherwise if hwtable.c is filled with a lot of duplicate entries,
> it could be unmanageable.
> 
Well.
MSA2000 is already out of support, and P2000 is heading there.
So it's _extremely_ unlikely that we get additional hardware entries for
those.
At the same time, re-grouping hardware entries by combining regular
expressions always has the real risk of messing up the regular
expressions, and requires a test on the actual hardware to check if we
didn't mess up. Seeing the both are basically unsupported it'll get hard
to validate this.
For this reasons I prefer to leave them alone.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Xose Vazquez Perez]

On 06/10/2016 08:17 AM, Hannes Reinecke wrote:

> On 06/09/2016 04:20 PM, Xose Vazquez Perez wrote:

>> In VxVM(libvxmsa2kfc_sa) they are grouped that way:
>> https://web.archive.org/web/20160608233140/https://sort.veritas.com/asl/details/756
>>
>> Otherwise if hwtable.c is filled with a lot of duplicate entries,
>> it could be unmanageable.

> Well.
> MSA2000 is already out of support, and P2000 is heading there.
> So it's _extremely_ unlikely that we get additional hardware entries for
> those.
> At the same time, re-grouping hardware entries by combining regular
> expressions always has the real risk of messing up the regular
> expressions, and requires a test on the actual hardware to check if we
> didn't mess up. Seeing the both are basically unsupported it'll get hard
> to validate this.
> For this reasons I prefer to leave them alone.

BTW, a "Unit serial number" attribute could be added to provide more flexibility.

Re: [PATCH 05/57] Add HP MSA 2040 to the hardware table

[Hannes Reinecke]

On 06/10/2016 04:30 PM, Xose Vazquez Perez wrote:
> On 06/10/2016 08:17 AM, Hannes Reinecke wrote:
>
>> On 06/09/2016 04:20 PM, Xose Vazquez Perez wrote:
>
>>> In VxVM(libvxmsa2kfc_sa) they are grouped that way:
>>> https://web.archive.org/web/20160608233140/https://sort.veritas.com/asl/details/756
>>>
>>> Otherwise if hwtable.c is filled with a lot of duplicate entries,
>>> it could be unmanageable.
>
>> Well.
>> MSA2000 is already out of support, and P2000 is heading there.
>> So it's _extremely_ unlikely that we get additional hardware entries for
>> those.
>> At the same time, re-grouping hardware entries by combining regular
>> expressions always has the real risk of messing up the regular
>> expressions, and requires a test on the actual hardware to check if we
>> didn't mess up. Seeing the both are basically unsupported it'll get hard
>> to validate this.
>> For this reasons I prefer to leave them alone.
>
> BTW, a "Unit serial number" attribute could be added to provide more flexibility.
>
Errm.

And this relates to the quoted mail like ... how?

Please send a separate mail (with a new subject) for this.
Otherwise we'll never find it again on the mailing list archives.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[PATCH 06/57] Use ALUA for HP 3PAR

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/hwtable.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
index 28ee595..4549268 100644
--- a/libmultipath/hwtable.c
+++ b/libmultipath/hwtable.c
@@ -65,13 +65,13 @@ static struct hwentry default_hw[] = {
 		.vendor        = "3PARdata",
 		.product       = "VV",
 		.features      = DEFAULT_FEATURES,
-		.hwhandler     = DEFAULT_HWHANDLER,
-		.pgpolicy      = MULTIBUS,
-		.pgfailback    = FAILBACK_UNDEF,
+		.hwhandler     = "1 alua",
+		.pgpolicy      = GROUP_BY_PRIO,
+		.pgfailback    = -FAILBACK_IMMEDIATE,
 		.rr_weight     = RR_WEIGHT_NONE,
-		.no_path_retry = NO_PATH_RETRY_UNDEF,
-		.checker_name  = DEFAULT_CHECKER,
-		.prio_name     = DEFAULT_PRIO,
+		.no_path_retry = 18,
+		.checker_name  = TUR,
+		.prio_name     = PRIO_ALUA,
 		.prio_args     = NULL,
 	},
 	{
-- 
2.6.6

Re: [PATCH 06/57] Use ALUA for HP 3PAR

[Sebastian Herbszt]

Hannes Reinecke wrote:
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/hwtable.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)

You missed to also change multipath.conf.defaults.
 
> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
> index 28ee595..4549268 100644
> --- a/libmultipath/hwtable.c
> +++ b/libmultipath/hwtable.c
> @@ -65,13 +65,13 @@ static struct hwentry default_hw[] = {
>  		.vendor        = "3PARdata",
>  		.product       = "VV",
>  		.features      = DEFAULT_FEATURES,
> -		.hwhandler     = DEFAULT_HWHANDLER,
> -		.pgpolicy      = MULTIBUS,
> -		.pgfailback    = FAILBACK_UNDEF,
> +		.hwhandler     = "1 alua",
> +		.pgpolicy      = GROUP_BY_PRIO,
> +		.pgfailback    = -FAILBACK_IMMEDIATE,
>  		.rr_weight     = RR_WEIGHT_NONE,
> -		.no_path_retry = NO_PATH_RETRY_UNDEF,
> -		.checker_name  = DEFAULT_CHECKER,
> -		.prio_name     = DEFAULT_PRIO,
> +		.no_path_retry = 18,
> +		.checker_name  = TUR,
> +		.prio_name     = PRIO_ALUA,
>  		.prio_args     = NULL,
>  	},
>  	{

Can you please elaborate on this?
I think the correct setting depends on the configured
"host persona". Currently the expected value would be
"Generic / 1". The patch changes it to "Generic-ALUA / 2".
Does the above change still work if the "host persona" is
not modified and still set to "Generic / 1"?

Sebastian

Re: [PATCH 06/57] Use ALUA for HP 3PAR

[Hannes Reinecke]

On 04/29/2016 12:25 AM, Sebastian Herbszt wrote:
> Hannes Reinecke wrote:
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  libmultipath/hwtable.c | 12 ++++++------
>>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> You missed to also change multipath.conf.defaults.
>  
>> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
>> index 28ee595..4549268 100644
>> --- a/libmultipath/hwtable.c
>> +++ b/libmultipath/hwtable.c
>> @@ -65,13 +65,13 @@ static struct hwentry default_hw[] = {
>>  		.vendor        = "3PARdata",
>>  		.product       = "VV",
>>  		.features      = DEFAULT_FEATURES,
>> -		.hwhandler     = DEFAULT_HWHANDLER,
>> -		.pgpolicy      = MULTIBUS,
>> -		.pgfailback    = FAILBACK_UNDEF,
>> +		.hwhandler     = "1 alua",
>> +		.pgpolicy      = GROUP_BY_PRIO,
>> +		.pgfailback    = -FAILBACK_IMMEDIATE,
>>  		.rr_weight     = RR_WEIGHT_NONE,
>> -		.no_path_retry = NO_PATH_RETRY_UNDEF,
>> -		.checker_name  = DEFAULT_CHECKER,
>> -		.prio_name     = DEFAULT_PRIO,
>> +		.no_path_retry = 18,
>> +		.checker_name  = TUR,
>> +		.prio_name     = PRIO_ALUA,
>>  		.prio_args     = NULL,
>>  	},
>>  	{
> 
> Can you please elaborate on this?
> I think the correct setting depends on the configured
> "host persona". Currently the expected value would be
> "Generic / 1". The patch changes it to "Generic-ALUA / 2".
> Does the above change still work if the "host persona" is
> not modified and still set to "Generic / 1"?
> 
I have never ever _seen_ an 3Par array, so I wasn't aware that you
could switch host 'persona'. So I have to rely on our partners (ie
HPE) for correct configuration here.
But given that fact we should better switch back to use the default
hw_handler and use the 'retain_attached_hw_handler' option.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 06/57] Use ALUA for HP 3PAR

[Sebastian Herbszt]

Hannes Reinecke wrote:
> On 04/29/2016 12:25 AM, Sebastian Herbszt wrote:
> > Hannes Reinecke wrote:
> >> Signed-off-by: Hannes Reinecke <hare@suse.de>
> >> ---
> >>  libmultipath/hwtable.c | 12 ++++++------
> >>  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > You missed to also change multipath.conf.defaults.
> >  
> >> diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
> >> index 28ee595..4549268 100644
> >> --- a/libmultipath/hwtable.c
> >> +++ b/libmultipath/hwtable.c
> >> @@ -65,13 +65,13 @@ static struct hwentry default_hw[] = {
> >>  		.vendor        = "3PARdata",
> >>  		.product       = "VV",
> >>  		.features      = DEFAULT_FEATURES,
> >> -		.hwhandler     = DEFAULT_HWHANDLER,
> >> -		.pgpolicy      = MULTIBUS,
> >> -		.pgfailback    = FAILBACK_UNDEF,
> >> +		.hwhandler     = "1 alua",
> >> +		.pgpolicy      = GROUP_BY_PRIO,
> >> +		.pgfailback    = -FAILBACK_IMMEDIATE,
> >>  		.rr_weight     = RR_WEIGHT_NONE,
> >> -		.no_path_retry = NO_PATH_RETRY_UNDEF,
> >> -		.checker_name  = DEFAULT_CHECKER,
> >> -		.prio_name     = DEFAULT_PRIO,
> >> +		.no_path_retry = 18,
> >> +		.checker_name  = TUR,
> >> +		.prio_name     = PRIO_ALUA,
> >>  		.prio_args     = NULL,
> >>  	},
> >>  	{
> > 
> > Can you please elaborate on this?
> > I think the correct setting depends on the configured
> > "host persona". Currently the expected value would be
> > "Generic / 1". The patch changes it to "Generic-ALUA / 2".
> > Does the above change still work if the "host persona" is
> > not modified and still set to "Generic / 1"?
> > 
> I have never ever _seen_ an 3Par array, so I wasn't aware that you
> could switch host 'persona'. So I have to rely on our partners (ie
> HPE) for correct configuration here.

HPE published at least two guides:

"HPE 3PAR SUSE Linux Enterprise Implementation Guide" [1] pp. 36-38
"HPE 3PAR Red Hat Enterprise Linux and Oracle Linux Implementation Guide" [2]
pp. 104-110

[1] http://h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=5383451&docId=emr_na-c02663748&docLocale=en_US
[2] http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04448818-9.pdf

Sebastian

[PATCH 07/57] Add LIO-ORG/SUSE RBD backend hardware defaults

[Hannes Reinecke]

The LIO-ORG RBD backend always supports ALUA, and should be assigned
the correct values for multipathing.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/hwtable.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/libmultipath/hwtable.c b/libmultipath/hwtable.c
index 4549268..e57a69a 100644
--- a/libmultipath/hwtable.c
+++ b/libmultipath/hwtable.c
@@ -1156,6 +1156,20 @@ static struct hwentry default_hw[] = {
 		.prio_args     = NULL,
 	},
 	{
+		.vendor	       = "(LIO-ORG|SUSE)",
+		.product       = "RBD",
+		.features      = DEFAULT_FEATURES,
+		.hwhandler     = "1 alua",
+		.pgpolicy      = GROUP_BY_PRIO,
+		.pgfailback    = -FAILBACK_IMMEDIATE,
+		.rr_weight     = RR_WEIGHT_NONE,
+		.no_path_retry = 12,
+		.checker_name  = TUR,
+		.minio         = 100,
+		.prio_name     = PRIO_ALUA,
+		.prio_args     = NULL,
+	},
+	{
 		.vendor	       = "DataCore",
 		.product       = "SANmelody",
 		.features      = DEFAULT_FEATURES,
-- 
2.6.6

Re: [PATCH 07/57] Add LIO-ORG/SUSE RBD backend hardware defaults

[Sebastian Herbszt]

Hannes Reinecke wrote:
> The LIO-ORG RBD backend always supports ALUA, and should be assigned
> the correct values for multipathing.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/hwtable.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)

You missed adding it to multipath.conf.defaults.

Sebastian

[PATCH 08/57] Allow for empty SCSI revision

[Hannes Reinecke]

Some (broken) SCSI devices return an empty SCSI revision, but
this doesn't indicate a broken device. So allow for an empty
revision string when discover devices.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/discovery.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index bd65354..446c91e 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -1095,7 +1095,7 @@ scsi_sysfs_pathinfo (struct path * pp)
 
 	condlog(3, "%s: product = %s", pp->dev, pp->product_id);
 
-	if (sysfs_get_rev(parent, pp->rev, SCSI_REV_SIZE) <= 0)
+	if (sysfs_get_rev(parent, pp->rev, SCSI_REV_SIZE) < 0)
 		return 1;
 
 	condlog(3, "%s: rev = %s", pp->dev, pp->rev);
-- 
2.6.6

[PATCH 09/57] libmultipath: Do not use 'sscanf' for parsing integers

[Hannes Reinecke]

'sscanf' doesn't really have a way of signalling errors,
so strtoul(l) should be used instead.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/discovery.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 446c91e..324e217 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -6,6 +6,7 @@
 #include <stdio.h>
 #include <ctype.h>
 #include <unistd.h>
+#include <limits.h>
 #include <fcntl.h>
 #include <sys/ioctl.h>
 #include <sys/stat.h>
@@ -237,8 +238,8 @@ sysfs_get_timeout(struct path *pp, unsigned int *timeout)
 	const char *attr = NULL;
 	const char *subsys;
 	struct udev_device *parent;
-	int r;
-	unsigned int t;
+	char *eptr;
+	unsigned long t;
 
 	if (!pp->udev || pp->bus != SYSFS_BUS_SCSI)
 		return -ENOSYS;
@@ -256,14 +257,17 @@ sysfs_get_timeout(struct path *pp, unsigned int *timeout)
 		return -ENXIO;
 	}
 
-	r = sscanf(attr, "%u\n", &t);
-
-	if (r != 1) {
+	t = strtoul(attr, &eptr, 0);
+	if (attr == eptr || t == ULONG_MAX) {
 		condlog(3, "%s: Cannot parse timeout attribute '%s'",
 			pp->dev, attr);
 		return -EINVAL;
 	}
-
+	if (t > UINT_MAX) {
+		condlog(3, "%s: Overflow in timeout value '%s'",
+			pp->dev, attr);
+		return -ERANGE;
+	}
 	*timeout = t;
 
 	return 0;
@@ -482,7 +486,7 @@ static void
 sysfs_set_rport_tmo(struct multipath *mpp, struct path *pp)
 {
 	struct udev_device *rport_dev = NULL;
-	char value[16];
+	char value[16], *eptr;
 	char rport_id[32];
 	unsigned long long tmo = 0;
 	int ret;
@@ -522,7 +526,8 @@ sysfs_set_rport_tmo(struct multipath *mpp, struct path *pp)
 				"error %d", rport_id, -ret);
 			goto out;
 		}
-		if (sscanf(value, "%llu\n", &tmo) != 1) {
+		tmo = strtoull(value, &eptr, 0);
+		if (value == eptr || tmo == ULLONG_MAX) {
 			condlog(0, "%s: Cannot parse dev_loss_tmo "
 				"attribute '%s'", rport_id, value);
 			goto out;
-- 
2.6.6

[PATCH 10/57] libmultipath: finally fix dev_loss_tmo setting

[Hannes Reinecke]

We need to take the current value when comparing
'dev_loss_tmo' and 'fast_io_fail', otherwise we still
might be getting an error as we might comparing wrong
values.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/defaults.h    |  1 +
 libmultipath/discovery.c   | 72 ++++++++++++++++++++++++----------------------
 multipath/multipath.conf.5 |  2 +-
 3 files changed, 39 insertions(+), 36 deletions(-)

diff --git a/libmultipath/defaults.h b/libmultipath/defaults.h
index bce8bcc..96f5a2c 100644
--- a/libmultipath/defaults.h
+++ b/libmultipath/defaults.h
@@ -15,6 +15,7 @@
 #define DEFAULT_REASSIGN_MAPS	0
 #define DEFAULT_FIND_MULTIPATHS	0
 #define DEFAULT_FAST_IO_FAIL	5
+#define DEFAULT_DEV_LOSS_TMO   600
 #define DEFAULT_RETAIN_HWHANDLER RETAIN_HWHANDLER_OFF
 #define DEFAULT_DETECT_PRIO DETECT_PRIO_OFF
 #define DEFAULT_DEFERRED_REMOVE DEFERRED_REMOVE_OFF
diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 324e217..83cc4f7 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -504,6 +504,22 @@ sysfs_set_rport_tmo(struct multipath *mpp, struct path *pp)
 		pp->sg_id.channel, pp->sg_id.scsi_id, rport_id);
 
 	/*
+	 * read the current dev_loss_tmo value from sysfs
+	 */
+	ret = sysfs_attr_get_value(rport_dev, "dev_loss_tmo", value, 16);
+	if (ret <= 0) {
+		condlog(0, "%s: failed to read dev_loss_tmo value, "
+			"error %d", rport_id, -ret);
+		goto out;
+	}
+	tmo = strtoull(value, &eptr, 0);
+	if (value == eptr || tmo == ULLONG_MAX) {
+		condlog(0, "%s: Cannot parse dev_loss_tmo "
+			"attribute '%s'", rport_id, value);
+		goto out;
+	}
+
+	/*
 	 * This is tricky.
 	 * dev_loss_tmo will be limited to 600 if fast_io_fail
 	 * is _not_ set.
@@ -514,45 +530,31 @@ sysfs_set_rport_tmo(struct multipath *mpp, struct path *pp)
 	 * then set fast_io_fail, and _then_ set dev_loss_tmo
 	 * to the correct value.
 	 */
-	memset(value, 0, 16);
 	if (mpp->fast_io_fail != MP_FAST_IO_FAIL_UNSET &&
 	    mpp->fast_io_fail != MP_FAST_IO_FAIL_ZERO &&
 	    mpp->fast_io_fail != MP_FAST_IO_FAIL_OFF) {
 		/* Check if we need to temporarily increase dev_loss_tmo */
-		ret = sysfs_attr_get_value(rport_dev, "dev_loss_tmo",
-					   value, 16);
-		if (ret <= 0) {
-			condlog(0, "%s: failed to read dev_loss_tmo value, "
-				"error %d", rport_id, -ret);
-			goto out;
-		}
-		tmo = strtoull(value, &eptr, 0);
-		if (value == eptr || tmo == ULLONG_MAX) {
-			condlog(0, "%s: Cannot parse dev_loss_tmo "
-				"attribute '%s'", rport_id, value);
-			goto out;
-		}
 		if (mpp->fast_io_fail >= tmo) {
+			/* Increase dev_loss_tmo temporarily */
 			snprintf(value, 16, "%u", mpp->fast_io_fail + 1);
+			ret = sysfs_attr_set_value(rport_dev, "dev_loss_tmo",
+						   value, strlen(value));
+			if (ret <= 0) {
+				if (ret == -EBUSY)
+					condlog(3, "%s: rport blocked",
+						rport_id);
+				else
+					condlog(0, "%s: failed to set "
+						"dev_loss_tmo to %s, error %d",
+						rport_id, value, -ret);
+				goto out;
+			}
 		}
-	} else if (mpp->dev_loss > 600) {
-		condlog(3, "%s: limiting dev_loss_tmo to 600, since "
-			"fast_io_fail is not set", rport_id);
-		snprintf(value, 16, "%u", 600);
-	} else {
-		snprintf(value, 16, "%u", mpp->dev_loss);
-	}
-	if (strlen(value)) {
-		ret = sysfs_attr_set_value(rport_dev, "dev_loss_tmo",
-					   value, strlen(value));
-		if (ret <= 0) {
-			if (ret == -EBUSY)
-				condlog(3, "%s: rport blocked", rport_id);
-			else
-				condlog(0, "%s: failed to set dev_loss_tmo to %s, error %d",
-					rport_id, value, -ret);
-			goto out;
-		}
+	} else if (mpp->dev_loss > DEFAULT_DEV_LOSS_TMO) {
+		condlog(3, "%s: limiting dev_loss_tmo to %d, since "
+			"fast_io_fail is not set",
+			rport_id, DEFAULT_DEV_LOSS_TMO);
+		mpp->dev_loss = DEFAULT_DEV_LOSS_TMO;
 	}
 	if (mpp->fast_io_fail != MP_FAST_IO_FAIL_UNSET) {
 		if (mpp->fast_io_fail == MP_FAST_IO_FAIL_OFF)
@@ -571,7 +573,7 @@ sysfs_set_rport_tmo(struct multipath *mpp, struct path *pp)
 					rport_id, value, -ret);
 		}
 	}
-	if (tmo > 0) {
+	if (mpp->dev_loss > 0) {
 		snprintf(value, 16, "%u", mpp->dev_loss);
 		ret = sysfs_attr_set_value(rport_dev, "dev_loss_tmo",
 					   value, strlen(value));
@@ -673,11 +675,11 @@ sysfs_set_scsi_tmo (struct multipath *mpp)
 			no_path_retry_tmo = MAX_DEV_LOSS_TMO;
 		if (no_path_retry_tmo > dev_loss_tmo)
 			dev_loss_tmo = no_path_retry_tmo;
-		condlog(3, "%s: update dev_loss_tmo to %d",
+		condlog(3, "%s: update dev_loss_tmo to %u",
 			mpp->alias, dev_loss_tmo);
 	} else if (mpp->no_path_retry == NO_PATH_RETRY_QUEUE) {
 		dev_loss_tmo = MAX_DEV_LOSS_TMO;
-		condlog(3, "%s: update dev_loss_tmo to %d",
+		condlog(3, "%s: update dev_loss_tmo to %u",
 			mpp->alias, dev_loss_tmo);
 	}
 	mpp->dev_loss = dev_loss_tmo;
diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5
index 0d4df0f..b21279e 100644
--- a/multipath/multipath.conf.5
+++ b/multipath/multipath.conf.5
@@ -394,7 +394,7 @@ retry interval
 if a number of retries is given with \fIno_path_retry\fR and the
 overall retry interval is longer than the specified \fIdev_loss_tmo\fR value.
 The linux kernel will cap this value to \fI300\fR if \fBfast_io_fail_tmo\fR
-is not set.
+is not set. Default is 600.
 .TP
 .B queue_without_daemon
 If set to
-- 
2.6.6

[PATCH 11/57] multipathd: fixup queueing mode in 'show maps status'

[Hannes Reinecke]

When manually disabling queueing via 'disablequeueing map $map'
the queueing status in 'show maps status' is not updated.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/print.c      | 6 ++++--
 multipathd/cli_handlers.c | 4 ++--
 multipathd/main.c         | 2 +-
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/libmultipath/print.c b/libmultipath/print.c
index 76bda93..96a1680 100644
--- a/libmultipath/print.c
+++ b/libmultipath/print.c
@@ -174,12 +174,14 @@ snprint_queueing (char * buff, size_t len, struct multipath * mpp)
 	else if (mpp->no_path_retry == NO_PATH_RETRY_UNDEF)
 		return snprintf(buff, len, "-");
 	else if (mpp->no_path_retry > 0) {
-		if (mpp->retry_tick)
+		if (mpp->retry_tick > 0)
 			return snprintf(buff, len, "%i sec",
 					mpp->retry_tick);
-		else
+		else if (mpp->retry_tick == 0)
 			return snprintf(buff, len, "%i chk",
 					mpp->no_path_retry);
+		else
+			return snprintf(buff, len, "off");
 	}
 	return 0;
 }
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 168b872..886a5fd 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -807,7 +807,7 @@ cli_disable_queueing(void *v, char **reply, int *len, void *data)
 		return 1;
 	}
 
-	mpp->retry_tick = 0;
+	mpp->retry_tick = -1;
 	dm_queue_if_no_path(mpp->alias, 0);
 	return 0;
 }
@@ -821,7 +821,7 @@ cli_disable_all_queueing(void *v, char **reply, int *len, void *data)
 
 	condlog(2, "disable queueing (operator)");
 	vector_foreach_slot(vecs->mpvec, mpp, i) {
-		mpp->retry_tick = 0;
+		mpp->retry_tick = -1;
 		dm_queue_if_no_path(mpp->alias, 0);
 	}
 	return 0;
diff --git a/multipathd/main.c b/multipathd/main.c
index 626f703..d9c0b8b 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1200,7 +1200,7 @@ retry_count_tick(vector mpvec)
 	unsigned int i;
 
 	vector_foreach_slot (mpvec, mpp, i) {
-		if (mpp->retry_tick) {
+		if (mpp->retry_tick > 0) {
 			mpp->stat_total_queueing_time++;
 			condlog(4, "%s: Retrying.. No active path", mpp->alias);
 			if(--mpp->retry_tick == 0) {
-- 
2.6.6

[PATCH 12/57] multipathd: fixup a crash when invoking CLI commands

[Hannes Reinecke]

The command buffer wasn't cleared, so the CLI might crash on startup.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/main.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/multipathd/main.c b/multipathd/main.c
index d9c0b8b..18b225a 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -2244,6 +2244,7 @@ main (int argc, char *argv[])
 
 		if (load_config(DEFAULT_CONFIGFILE, udev_new()))
 			exit(1);
+		memset(cmd, 0x0, CMDSIZE);
 		while (optind < argc) {
 			if (strchr(argv[optind], ' '))
 				c += snprintf(c, s + CMDSIZE - c, "\"%s\" ", argv[optind]);
-- 
2.6.6

[PATCH 13/57] multipathd: print error message for invalid arguments

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/main.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 18b225a..59f0c68 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -2234,7 +2234,9 @@ main (int argc, char *argv[])
 			conf->ignore_new_devs = 1;
 			break;
 		default:
-			;
+			fprintf(stderr, "Invalid argument '-%c'\n",
+				optopt);
+			exit(1);
 		}
 	}
 	if (optind < argc) {
-- 
2.6.6

[PATCH 14/57] libmultipath: correctly initialize pp->sg_id

[Hannes Reinecke]

The default SCSI protocol is 'SCSI_PROTOCOL_UNSPEC';
'0' is SCSI_PROTOCOL_FCP.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/structs.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/libmultipath/structs.c b/libmultipath/structs.c
index 0538327..b773d1c 100644
--- a/libmultipath/structs.c
+++ b/libmultipath/structs.c
@@ -94,6 +94,7 @@ alloc_path (void)
 		pp->sg_id.channel = -1;
 		pp->sg_id.scsi_id = -1;
 		pp->sg_id.lun = -1;
+		pp->sg_id.proto_id = SCSI_PROTOCOL_UNSPEC;
 		pp->fd = -1;
 		pp->priority = PRIO_UNDEF;
 	}
-- 
2.6.6

[PATCH 15/57] libmultipath: correctly display checker status

[Hannes Reinecke]

The '%d chk' message should only be used if there are active paths,
otherwise the path checker most certainly haven't been run.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/print.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libmultipath/print.c b/libmultipath/print.c
index 96a1680..7fec6e9 100644
--- a/libmultipath/print.c
+++ b/libmultipath/print.c
@@ -177,7 +177,7 @@ snprint_queueing (char * buff, size_t len, struct multipath * mpp)
 		if (mpp->retry_tick > 0)
 			return snprintf(buff, len, "%i sec",
 					mpp->retry_tick);
-		else if (mpp->retry_tick == 0)
+		else if (mpp->retry_tick == 0 && mpp->nr_active > 0)
 			return snprintf(buff, len, "%i chk",
 					mpp->no_path_retry);
 		else
-- 
2.6.6

[PATCH 16/57] libmultipath: call get_uid() for all paths

[Hannes Reinecke]

When we're reading the device UID off uevents we don't have to
check the path state as we're not actually doing I/O.
So pass in the path_state to get_uid() and evaluate it only
if ->getuid is selected.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/discovery.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 83cc4f7..2fa0907 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -1513,7 +1513,7 @@ get_vpd_uid(struct path * pp)
 }
 
 static int
-get_uid (struct path * pp)
+get_uid (struct path * pp, int path_state)
 {
 	char *c;
 	const char *origin = "unknown";
@@ -1533,13 +1533,14 @@ get_uid (struct path * pp)
 
 		/* Use 'getuid' callout, deprecated */
 		condlog(1, "%s: using deprecated getuid callout", pp->dev);
-		if (apply_format(pp->getuid, &buff[0], pp)) {
+		if (path_state != PATH_UP) {
+			condlog(3, "%s: path inaccessible", pp->dev);
+			len = -EWOULDBLOCK;
+		} else if (apply_format(pp->getuid, &buff[0], pp)) {
 			condlog(0, "error formatting uid callout command");
-			memset(pp->wwid, 0, WWID_SIZE);
 			len = -EINVAL;
 		} else if (execute_program(buff, pp->wwid, WWID_SIZE)) {
 			condlog(3, "error calling out %s", buff);
-			memset(pp->wwid, 0, WWID_SIZE);
 			len = -EIO;
 		} else
 			len = strlen(pp->wwid);
@@ -1559,7 +1560,7 @@ get_uid (struct path * pp)
 			len = get_vpd_uid(pp);
 			origin = "sysfs";
 			pp->uid_attribute = NULL;
-			if (len < 0) {
+			if (len < 0 && path_state == PATH_UP) {
 				condlog(1, "%s: failed to get sysfs uid: %s",
 					pp->dev, strerror(-len));
 				len = get_vpd_sgio(pp->fd, 0x83, pp->wwid,
@@ -1659,7 +1660,7 @@ pathinfo (struct path *pp, vector hwtable, int mask)
 	}
 
 	if ((mask & DI_WWID) && !strlen(pp->wwid)) {
-		get_uid(pp);
+		get_uid(pp, path_state);
 		if (!strlen(pp->wwid)) {
 			pp->initialized = INIT_MISSING_UDEV;
 			pp->tick = conf->retrigger_delay;
-- 
2.6.6

[PATCH 17/57] libmultipath: additional logging messages when formatting callout

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/callout.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/libmultipath/callout.c b/libmultipath/callout.c
index 4cbbd06..d671b0c 100644
--- a/libmultipath/callout.c
+++ b/libmultipath/callout.c
@@ -201,8 +201,10 @@ apply_format (char * string, char * cmd, struct path * pp)
 	}
 	pos++;
 
-	if (!*pos)
+	if (!*pos) {
+		condlog(3, "formatted callout = %s", dst);
 		return 0;
+	}
 
 	len = strlen(pos) + 1;
 	myfree -= len;
@@ -211,6 +213,6 @@ apply_format (char * string, char * cmd, struct path * pp)
 		return 1;
 
 	snprintf(p, len, "%s", pos);
-	condlog(3, "reformated callout = %s", dst);
+	condlog(3, "reformatted callout = %s", dst);
 	return 0;
 }
-- 
2.6.6

[PATCH 18/57] libmpathpersist: Fixup whitespaces in Makefile

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmpathpersist/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libmpathpersist/Makefile b/libmpathpersist/Makefile
index e49cdb9..59db73a 100644
--- a/libmpathpersist/Makefile
+++ b/libmpathpersist/Makefile
@@ -40,8 +40,8 @@ install: $(LIBS)
 
 uninstall:
 	rm -f $(DESTDIR)$(syslibdir)/$(LIBS)
-	rm $(DESTDIR)$(man3dir)/mpath_persistent_reserve_in.3.gz	
-	rm $(DESTDIR)$(man3dir)/mpath_persistent_reserve_out.3.gz	
+	rm $(DESTDIR)$(man3dir)/mpath_persistent_reserve_in.3.gz
+	rm $(DESTDIR)$(man3dir)/mpath_persistent_reserve_out.3.gz
 
 clean:
 	rm -f core *.a *.o 
-- 
2.6.6

[PATCH 19/57] multipathd: Do not print misleading message 'not found in pathvec'

[Hannes Reinecke]

When looking up a path in the existing configuration it is perfectly
possible for the path not to be present.
This should not generate a message as it might be errorneously
interpreted as an error.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/structs.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/libmultipath/structs.c b/libmultipath/structs.c
index b773d1c..39b2e05 100644
--- a/libmultipath/structs.c
+++ b/libmultipath/structs.c
@@ -404,7 +404,6 @@ find_path_by_dev (vector pathvec, char * dev)
 		if (!strcmp(pp->dev, dev))
 			return pp;
 
-	condlog(3, "%s: not found in pathvec", dev);
 	return NULL;
 }
 
@@ -421,7 +420,6 @@ find_path_by_devt (vector pathvec, char * dev_t)
 		if (!strcmp(pp->dev_t, dev_t))
 			return pp;
 
-	condlog(3, "%s: not found in pathvec", dev_t);
 	return NULL;
 }
 
-- 
2.6.6

Re: [PATCH 19/57] multipathd: Do not print misleading message 'not found in pathvec'

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:20PM +0200, Hannes Reinecke wrote:
> When looking up a path in the existing configuration it is perfectly
> possible for the path not to be present.
> This should not generate a message as it might be errorneously
> interpreted as an error.
> 

Do you feel really strongly that these messages should go? I don't think
that they are essential, but they can be useful when debugging an issue
to see the route the code took. We could move them to level 4, but
personally, I find the amount of messages generated at log level 4 to be
so high that it's a pain to use to track bugs that don't occur right
away. Another possibility would be to change the message to something
more innocuous.

Thoughts?

-Ben

> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/structs.c | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/libmultipath/structs.c b/libmultipath/structs.c
> index b773d1c..39b2e05 100644
> --- a/libmultipath/structs.c
> +++ b/libmultipath/structs.c
> @@ -404,7 +404,6 @@ find_path_by_dev (vector pathvec, char * dev)
>  		if (!strcmp(pp->dev, dev))
>  			return pp;
>  
> -	condlog(3, "%s: not found in pathvec", dev);
>  	return NULL;
>  }
>  
> @@ -421,7 +420,6 @@ find_path_by_devt (vector pathvec, char * dev_t)
>  		if (!strcmp(pp->dev_t, dev_t))
>  			return pp;
>  
> -	condlog(3, "%s: not found in pathvec", dev_t);
>  	return NULL;
>  }
>  
> -- 
> 2.6.6

Re: [PATCH 19/57] multipathd: Do not print misleading message 'not found in pathvec'

[Hannes Reinecke]

On 05/02/2016 05:40 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:20PM +0200, Hannes Reinecke wrote:
>> When looking up a path in the existing configuration it is perfectly
>> possible for the path not to be present.
>> This should not generate a message as it might be errorneously
>> interpreted as an error.
>>
> 
> Do you feel really strongly that these messages should go? I don't think
> that they are essential, but they can be useful when debugging an issue
> to see the route the code took. We could move them to level 4, but
> personally, I find the amount of messages generated at log level 4 to be
> so high that it's a pain to use to track bugs that don't occur right
> away. Another possibility would be to change the message to something
> more innocuous.
> 
I'm not particulary attached to this patch.
The main issues I have is that we're generating two identical messages
at two different locations (ie making it hard to debug), and that the
message don't indicate whether it's an error or not.

But sure, I can easily drop this patch.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 19/57] multipathd: Do not print misleading message 'not found in pathvec'

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 1614 bytes --]

The patch was merged as-is, so reinstating the warning, using another
non-conflicting phrasing, would require a new patch.

On Tue, May 3, 2016 at 7:47 AM, Hannes Reinecke <hare@suse.de> wrote:

> On 05/02/2016 05:40 PM, Benjamin Marzinski wrote:
> > On Wed, Apr 27, 2016 at 01:10:20PM +0200, Hannes Reinecke wrote:
> >> When looking up a path in the existing configuration it is perfectly
> >> possible for the path not to be present.
> >> This should not generate a message as it might be errorneously
> >> interpreted as an error.
> >>
> >
> > Do you feel really strongly that these messages should go? I don't think
> > that they are essential, but they can be useful when debugging an issue
> > to see the route the code took. We could move them to level 4, but
> > personally, I find the amount of messages generated at log level 4 to be
> > so high that it's a pain to use to track bugs that don't occur right
> > away. Another possibility would be to change the message to something
> > more innocuous.
> >
> I'm not particulary attached to this patch.
> The main issues I have is that we're generating two identical messages
> at two different locations (ie making it hard to debug), and that the
> message don't indicate whether it's an error or not.
>
> But sure, I can easily drop this patch.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                   zSeries & Storage
> hare@suse.de                          +49 911 74053 688
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
>

[-- Attachment #1.2: Type: text/html, Size: 2253 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH 20/57] multipathd: Do not update the paths vec when removing paths

[Hannes Reinecke]

When we remove a path it's totally pointless to add it to
the path list first; it'll be removed on the next step anyway.
And we should be cleaning up the comments while we're at it.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/main.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 59f0c68..61b82f6 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -695,14 +695,8 @@ ev_remove_path (struct path *pp, struct vectors * vecs)
 	 */
 	if ((mpp = pp->mpp)) {
 		/*
-		 * transform the mp->pg vector of vectors of paths
-		 * into a mp->params string to feed the device-mapper
+		 * Remove path from paths list
 		 */
-		if (update_mpp_paths(mpp, vecs->pathvec)) {
-			condlog(0, "%s: failed to update paths",
-				mpp->alias);
-			goto fail;
-		}
 		if ((i = find_slot(mpp->paths, (void *)pp)) != -1)
 			vector_del_slot(mpp->paths, i);
 
@@ -735,6 +729,10 @@ ev_remove_path (struct path *pp, struct vectors * vecs)
 			 */
 		}
 
+		/*
+		 * transform the mp->pg vector of vectors of paths
+		 * into a mp->params string to feed the device-mapper
+		 */
 		if (setup_map(mpp, params, PARAMS_SIZE)) {
 			condlog(0, "%s: failed to setup map for"
 				" removal of path %s", mpp->alias, pp->dev);
@@ -1016,6 +1014,7 @@ uxlsnrloop (void * ap)
 
 	umask(077);
 	uxsock_listen(&uxsock_trigger, ap);
+	condlog(1, "terminate uxsock listener");
 
 	return NULL;
 }
-- 
2.6.6

Re: [PATCH 20/57] multipathd: Do not update the paths vec when removing paths

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:21PM +0200, Hannes Reinecke wrote:
> When we remove a path it's totally pointless to add it to
> the path list first; it'll be removed on the next step anyway.
> And we should be cleaning up the comments while we're at it.

This one causes problems. The easiest way to see that is to run
something like

# multipathd reload map <map> ; multipathd del path <path-in-that-map>

This really messes things up. The reason is that at the start of
ev_remove_path, there is no guarantee that any of the paths will be in
mpp->paths.  This is because when multipath runs the pgpolicyfn in
setup_map(), all of policy functions free mpp->paths once they have set
up the path groups.  I assume that this was done so that there is no
chance that the list of paths in mpp->paths will get out of sync with
the list of paths in the pathgroups.

I can see why it someone might want to only keep mpp->pg as the
definitive list of paths, and to use update_mpp_paths() to regenerate
mpp->paths when necessary. But that's not what multipathd does. Instead,
mpp->paths is almost always regenerated by calling setup_multipath()
later in the same function that called setup_map(). However not every
function will always do this.  ev_remove_path doesn't do this if domap()
fails, and reload_map() never calls setup_multipath(). coalesce_paths()
doesn't call setup_multipath() itself, but some if it's callers do. Even
if mpp->paths isn't restored right away, it will be when check_path
calls update_multipath_strings().

So, if you call "cli reload map" and then call "cli del path" before the
checker function restores mpp->paths, and multipath doesn't call
update_mpp_pths() in ev_remove_path, you get into problems.

The question is, what's the right thing to do?

Option 1 is to never delete mpp->paths in the first place. Then we can
probably do away with some more of the update_mpp_paths() calls. We just
need to make certain that whenever we update mpp->pg, we are always
either getting the paths from mpp->paths, or we call update_mpp_paths()
afterwards to sync them.

Option 2 is to say that we will alway regenerate mpp->paths whenever we
need it. In that case, we should probably be freeing it after we're done
with it.

I don't really care either way, as long as we're consistent. Otherwise
we'll get into bizzare situations like the one above.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  multipathd/main.c | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 59f0c68..61b82f6 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -695,14 +695,8 @@ ev_remove_path (struct path *pp, struct vectors * vecs)
>  	 */
>  	if ((mpp = pp->mpp)) {
>  		/*
> -		 * transform the mp->pg vector of vectors of paths
> -		 * into a mp->params string to feed the device-mapper
> +		 * Remove path from paths list
>  		 */
> -		if (update_mpp_paths(mpp, vecs->pathvec)) {
> -			condlog(0, "%s: failed to update paths",
> -				mpp->alias);
> -			goto fail;
> -		}
>  		if ((i = find_slot(mpp->paths, (void *)pp)) != -1)
>  			vector_del_slot(mpp->paths, i);
>  
> @@ -735,6 +729,10 @@ ev_remove_path (struct path *pp, struct vectors * vecs)
>  			 */
>  		}
>  
> +		/*
> +		 * transform the mp->pg vector of vectors of paths
> +		 * into a mp->params string to feed the device-mapper
> +		 */
>  		if (setup_map(mpp, params, PARAMS_SIZE)) {
>  			condlog(0, "%s: failed to setup map for"
>  				" removal of path %s", mpp->alias, pp->dev);
> @@ -1016,6 +1014,7 @@ uxlsnrloop (void * ap)
>  
>  	umask(077);
>  	uxsock_listen(&uxsock_trigger, ap);
> +	condlog(1, "terminate uxsock listener");
>  
>  	return NULL;
>  }
> -- 
> 2.6.6

Re: [PATCH 20/57] multipathd: Do not update the paths vec when removing paths

[Hannes Reinecke]

On 04/30/2016 12:39 AM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:21PM +0200, Hannes Reinecke wrote:
>> When we remove a path it's totally pointless to add it to
>> the path list first; it'll be removed on the next step anyway.
>> And we should be cleaning up the comments while we're at it.
> 
> This one causes problems. The easiest way to see that is to run
> something like
> 
> # multipathd reload map <map> ; multipathd del path <path-in-that-map>
> 
> This really messes things up. The reason is that at the start of
> ev_remove_path, there is no guarantee that any of the paths will be in
> mpp->paths.  This is because when multipath runs the pgpolicyfn in
> setup_map(), all of policy functions free mpp->paths once they have set
> up the path groups.  I assume that this was done so that there is no
> chance that the list of paths in mpp->paths will get out of sync with
> the list of paths in the pathgroups.
> 
> I can see why it someone might want to only keep mpp->pg as the
> definitive list of paths, and to use update_mpp_paths() to regenerate
> mpp->paths when necessary. But that's not what multipathd does. Instead,
> mpp->paths is almost always regenerated by calling setup_multipath()
> later in the same function that called setup_map(). However not every
> function will always do this.  ev_remove_path doesn't do this if domap()
> fails, and reload_map() never calls setup_multipath(). coalesce_paths()
> doesn't call setup_multipath() itself, but some if it's callers do. Even
> if mpp->paths isn't restored right away, it will be when check_path
> calls update_multipath_strings().
> 
> So, if you call "cli reload map" and then call "cli del path" before the
> checker function restores mpp->paths, and multipath doesn't call
> update_mpp_pths() in ev_remove_path, you get into problems.
> 
> The question is, what's the right thing to do?
> 
> Option 1 is to never delete mpp->paths in the first place. Then we can
> probably do away with some more of the update_mpp_paths() calls. We just
> need to make certain that whenever we update mpp->pg, we are always
> either getting the paths from mpp->paths, or we call update_mpp_paths()
> afterwards to sync them.
> 
> Option 2 is to say that we will alway regenerate mpp->paths whenever we
> need it. In that case, we should probably be freeing it after we're done
> with it.
> 
> I don't really care either way, as long as we're consistent. Otherwise
> we'll get into bizzare situations like the one above.
> 
Personally, I would opt for 1).
Regenerating mpp->paths has the very big risk of running into even
more race conditions, and I'd rather have it stable as far as possible.

Thanks for the clarification here.

I'll see to come up with a replacement patch.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 20/57] multipathd: Do not update the paths vec when removing paths

[Benjamin Marzinski]

On Mon, May 02, 2016 at 07:48:50AM +0200, Hannes Reinecke wrote:
> On 04/30/2016 12:39 AM, Benjamin Marzinski wrote:
> > On Wed, Apr 27, 2016 at 01:10:21PM +0200, Hannes Reinecke wrote:
> >> When we remove a path it's totally pointless to add it to
> >> the path list first; it'll be removed on the next step anyway.
> >> And we should be cleaning up the comments while we're at it.
> > 
> > This one causes problems. The easiest way to see that is to run
> > something like
> > 
> > # multipathd reload map <map> ; multipathd del path <path-in-that-map>
> > 
> > This really messes things up. The reason is that at the start of
> > ev_remove_path, there is no guarantee that any of the paths will be in
> > mpp->paths.  This is because when multipath runs the pgpolicyfn in
> > setup_map(), all of policy functions free mpp->paths once they have set
> > up the path groups.  I assume that this was done so that there is no
> > chance that the list of paths in mpp->paths will get out of sync with
> > the list of paths in the pathgroups.
> > 
> > I can see why it someone might want to only keep mpp->pg as the
> > definitive list of paths, and to use update_mpp_paths() to regenerate
> > mpp->paths when necessary. But that's not what multipathd does. Instead,
> > mpp->paths is almost always regenerated by calling setup_multipath()
> > later in the same function that called setup_map(). However not every
> > function will always do this.  ev_remove_path doesn't do this if domap()
> > fails, and reload_map() never calls setup_multipath(). coalesce_paths()
> > doesn't call setup_multipath() itself, but some if it's callers do. Even
> > if mpp->paths isn't restored right away, it will be when check_path
> > calls update_multipath_strings().
> > 
> > So, if you call "cli reload map" and then call "cli del path" before the
> > checker function restores mpp->paths, and multipath doesn't call
> > update_mpp_pths() in ev_remove_path, you get into problems.
> > 
> > The question is, what's the right thing to do?
> > 
> > Option 1 is to never delete mpp->paths in the first place. Then we can
> > probably do away with some more of the update_mpp_paths() calls. We just
> > need to make certain that whenever we update mpp->pg, we are always
> > either getting the paths from mpp->paths, or we call update_mpp_paths()
> > afterwards to sync them.
> > 
> > Option 2 is to say that we will alway regenerate mpp->paths whenever we
> > need it. In that case, we should probably be freeing it after we're done
> > with it.
> > 
> > I don't really care either way, as long as we're consistent. Otherwise
> > we'll get into bizzare situations like the one above.
> > 
> Personally, I would opt for 1).
> Regenerating mpp->paths has the very big risk of running into even
> more race conditions, and I'd rather have it stable as far as possible.

I lean towards option 1 too. Alos, we should probably make sure that we
call setup_multipath in all the code paths that call setup_map and domap
for consistency.

-Ben

> 
> Thanks for the clarification here.
> 
> I'll see to come up with a replacement patch.
> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		   Teamlead Storage & Networking
> hare@suse.de			               +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

Re: [PATCH 20/57] multipathd: Do not update the paths vec when removing paths

[Hannes Reinecke]

On 05/02/2016 05:12 PM, Benjamin Marzinski wrote:
> On Mon, May 02, 2016 at 07:48:50AM +0200, Hannes Reinecke wrote:
>> On 04/30/2016 12:39 AM, Benjamin Marzinski wrote:
>>> On Wed, Apr 27, 2016 at 01:10:21PM +0200, Hannes Reinecke wrote:
>>>> When we remove a path it's totally pointless to add it to
>>>> the path list first; it'll be removed on the next step anyway.
>>>> And we should be cleaning up the comments while we're at it.
>>>
>>> This one causes problems. The easiest way to see that is to run
>>> something like
>>>
>>> # multipathd reload map <map> ; multipathd del path <path-in-that-map>
>>>
>>> This really messes things up. The reason is that at the start of
>>> ev_remove_path, there is no guarantee that any of the paths will be in
>>> mpp->paths.  This is because when multipath runs the pgpolicyfn in
>>> setup_map(), all of policy functions free mpp->paths once they have set
>>> up the path groups.  I assume that this was done so that there is no
>>> chance that the list of paths in mpp->paths will get out of sync with
>>> the list of paths in the pathgroups.
>>>
>>> I can see why it someone might want to only keep mpp->pg as the
>>> definitive list of paths, and to use update_mpp_paths() to regenerate
>>> mpp->paths when necessary. But that's not what multipathd does. Instead,
>>> mpp->paths is almost always regenerated by calling setup_multipath()
>>> later in the same function that called setup_map(). However not every
>>> function will always do this.  ev_remove_path doesn't do this if domap()
>>> fails, and reload_map() never calls setup_multipath(). coalesce_paths()
>>> doesn't call setup_multipath() itself, but some if it's callers do. Even
>>> if mpp->paths isn't restored right away, it will be when check_path
>>> calls update_multipath_strings().
>>>
>>> So, if you call "cli reload map" and then call "cli del path" before the
>>> checker function restores mpp->paths, and multipath doesn't call
>>> update_mpp_pths() in ev_remove_path, you get into problems.
>>>
>>> The question is, what's the right thing to do?
>>>
>>> Option 1 is to never delete mpp->paths in the first place. Then we can
>>> probably do away with some more of the update_mpp_paths() calls. We just
>>> need to make certain that whenever we update mpp->pg, we are always
>>> either getting the paths from mpp->paths, or we call update_mpp_paths()
>>> afterwards to sync them.
>>>
>>> Option 2 is to say that we will alway regenerate mpp->paths whenever we
>>> need it. In that case, we should probably be freeing it after we're done
>>> with it.
>>>
>>> I don't really care either way, as long as we're consistent. Otherwise
>>> we'll get into bizzare situations like the one above.
>>>
>> Personally, I would opt for 1).
>> Regenerating mpp->paths has the very big risk of running into even
>> more race conditions, and I'd rather have it stable as far as possible.
> 
> I lean towards option 1 too. Alos, we should probably make sure that we
> call setup_multipath in all the code paths that call setup_map and domap
> for consistency.
> 
Actually, I've been looking at the code; and I think it might be easier
to make mpp->paths a local variable and remove it from struct mpp.
That way we can be sure no stale ->paths variable is left, and it will
always be computed correctly.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[PATCH 21/57] libmultipath: avoid double semicolon in lock.h

[Hannes Reinecke]

The definitions for lock() already have a semicolon, resulting
in a double semicolon when using the definitions like a normal
statement.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/lock.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libmultipath/lock.h b/libmultipath/lock.h
index 04ef78d..97af0f4 100644
--- a/libmultipath/lock.h
+++ b/libmultipath/lock.h
@@ -21,11 +21,11 @@ struct mutex_lock {
 	a.depth--; pthread_mutex_unlock(a.mutex)
 #define lock_cleanup_pop(a) \
 		fprintf(stderr, "%s:%s(%i) unlock %p depth: %d (%ld)\n", __FILE__, __FUNCTION__, __LINE__, a.mutex, a.depth, pthread_self()); \
-	pthread_cleanup_pop(1);
+	pthread_cleanup_pop(1)
 #else
 #define lock(a) a.depth++; pthread_mutex_lock(a.mutex)
 #define unlock(a) a.depth--; pthread_mutex_unlock(a.mutex)
-#define lock_cleanup_pop(a) pthread_cleanup_pop(1);
+#define lock_cleanup_pop(a) pthread_cleanup_pop(1)
 #endif
 
 void cleanup_lock (void * data);
-- 
2.6.6

[PATCH 22/57] multipath: use option '-i' when called from udev

[Hannes Reinecke]

multipath should be using the option '-i' to ignore the wwids
file when called from udev. Otherwise we might run into a race
condition with systemd and the system might not boot up correctly.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipath/multipath.rules | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipath/multipath.rules b/multipath/multipath.rules
index c8fb7e6..6b0d1cd 100644
--- a/multipath/multipath.rules
+++ b/multipath/multipath.rules
@@ -15,7 +15,7 @@ ENV{MPATH_SBIN_PATH}="/sbin"
 TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin"
 
 ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \
-	PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \
+	PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -i -u %k", \
 	ENV{DM_MULTIPATH_DEVICE_PATH}="1", ENV{ID_FS_TYPE}="none", \
 	ENV{SYSTEMD_READY}="0"
 
-- 
2.6.6

Re: [PATCH 22/57] multipath: use option '-i' when called from udev

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:23PM +0200, Hannes Reinecke wrote:
> multipath should be using the option '-i' to ignore the wwids
> file when called from udev. Otherwise we might run into a race
> condition with systemd and the system might not boot up correctly.

The race condition being? Are you talking about the udev rules not
claiming a path for multipath, but then having multipathd create a
multipath device using that path? I agree that this can be an issue.
Another way to solve it is to run mutipathd with the -n option in the
initramfs (see commit a8efa5838cf215febd853f282c26af62c0daa862). That
commit solves the race by making mutipathd ignore devices that aren't
already in the wwids file.  This also keeps the initramfs from picking a
different user_friendly_name than will be picked in late boot (but
hopefully that issue has been sorted out by other patches already).

I'm not NAKing this. The question of what to do about the differnces in
the distribution's systemd and udev configurations can be hashed out
outside of patch reviewing. I just want to make sure I understand the
race this is solving.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  multipath/multipath.rules | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/multipath/multipath.rules b/multipath/multipath.rules
> index c8fb7e6..6b0d1cd 100644
> --- a/multipath/multipath.rules
> +++ b/multipath/multipath.rules
> @@ -15,7 +15,7 @@ ENV{MPATH_SBIN_PATH}="/sbin"
>  TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin"
>  
>  ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \
> -	PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \
> +	PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -i -u %k", \
>  	ENV{DM_MULTIPATH_DEVICE_PATH}="1", ENV{ID_FS_TYPE}="none", \
>  	ENV{SYSTEMD_READY}="0"
>  
> -- 
> 2.6.6

Re: [PATCH 22/57] multipath: use option '-i' when called from udev

[Hannes Reinecke]

On 05/02/2016 05:31 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:23PM +0200, Hannes Reinecke wrote:
>> multipath should be using the option '-i' to ignore the wwids
>> file when called from udev. Otherwise we might run into a race
>> condition with systemd and the system might not boot up correctly.
> 
> The race condition being? Are you talking about the udev rules not
> claiming a path for multipath, but then having multipathd create a
> multipath device using that path? I agree that this can be an issue.
> Another way to solve it is to run mutipathd with the -n option in the
> initramfs (see commit a8efa5838cf215febd853f282c26af62c0daa862). That
> commit solves the race by making mutipathd ignore devices that aren't
> already in the wwids file.  This also keeps the initramfs from picking a
> different user_friendly_name than will be picked in late boot (but
> hopefully that issue has been sorted out by other patches already).
> 
This is in fact SUSE-specific, as we do _not_ require any a-priory
configuration. So we will start out with a system without any wwid file,
_but_ expect multipath to start up properly.
As the wwid file will only be created by multipathd the udev check will
always being false, and none of the multipath devices will be created.

> I'm not NAKing this. The question of what to do about the differnces in
> the distribution's systemd and udev configurations can be hashed out
> outside of patch reviewing. I just want to make sure I understand the
> race this is solving.
> 
This is in fact a good question.
We should be coming up with a common approach which would suit both needs.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 22/57] multipath: use option '-i' when called from udev

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 2207 bytes --]

Ben,

does this patch actually break the Red Hat integration ?
I'm inclined to merge it if not.

Thanks,
Christophe

On Tue, May 3, 2016 at 7:44 AM, Hannes Reinecke <hare@suse.de> wrote:

> On 05/02/2016 05:31 PM, Benjamin Marzinski wrote:
> > On Wed, Apr 27, 2016 at 01:10:23PM +0200, Hannes Reinecke wrote:
> >> multipath should be using the option '-i' to ignore the wwids
> >> file when called from udev. Otherwise we might run into a race
> >> condition with systemd and the system might not boot up correctly.
> >
> > The race condition being? Are you talking about the udev rules not
> > claiming a path for multipath, but then having multipathd create a
> > multipath device using that path? I agree that this can be an issue.
> > Another way to solve it is to run mutipathd with the -n option in the
> > initramfs (see commit a8efa5838cf215febd853f282c26af62c0daa862). That
> > commit solves the race by making mutipathd ignore devices that aren't
> > already in the wwids file.  This also keeps the initramfs from picking a
> > different user_friendly_name than will be picked in late boot (but
> > hopefully that issue has been sorted out by other patches already).
> >
> This is in fact SUSE-specific, as we do _not_ require any a-priory
> configuration. So we will start out with a system without any wwid file,
> _but_ expect multipath to start up properly.
> As the wwid file will only be created by multipathd the udev check will
> always being false, and none of the multipath devices will be created.
>
> > I'm not NAKing this. The question of what to do about the differnces in
> > the distribution's systemd and udev configurations can be hashed out
> > outside of patch reviewing. I just want to make sure I understand the
> > race this is solving.
> >
> This is in fact a good question.
> We should be coming up with a common approach which would suit both needs.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                   zSeries & Storage
> hare@suse.de                          +49 911 74053 688
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
>

[-- Attachment #1.2: Type: text/html, Size: 2951 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH 22/57] multipath: use option '-i' when called from udev

[Benjamin Marzinski]

On Tue, May 03, 2016 at 08:37:08AM +0200, Christophe Varoqui wrote:
>    Ben,
>    does this patch actually break the Red Hat integration ?
>    I'm inclined to merge it if not.

Like we talked about earlier, RedHat uses a completely different
multipath.rules file, which includes some command options that were
NAK'ed for inclusion. We can sort that out. Either we will or we won't
come to an agreement on how multipath should interact with udev. If we
don't we can probably just keep different rules files for the different
distributions. But having a mostly-but-not-quite working version of the
SUSE rules file really benefits nobody. I'm in favor of this going it.

-Ben

>    Thanks,
>    Christophe
>    On Tue, May 3, 2016 at 7:44 AM, Hannes Reinecke <[1]hare@suse.de> wrote:
> 
>      On 05/02/2016 05:31 PM, Benjamin Marzinski wrote:
>      > On Wed, Apr 27, 2016 at 01:10:23PM +0200, Hannes Reinecke wrote:
>      >> multipath should be using the option '-i' to ignore the wwids
>      >> file when called from udev. Otherwise we might run into a race
>      >> condition with systemd and the system might not boot up correctly.
>      >
>      > The race condition being? Are you talking about the udev rules not
>      > claiming a path for multipath, but then having multipathd create a
>      > multipath device using that path? I agree that this can be an issue.
>      > Another way to solve it is to run mutipathd with the -n option in the
>      > initramfs (see commit a8efa5838cf215febd853f282c26af62c0daa862). That
>      > commit solves the race by making mutipathd ignore devices that aren't
>      > already in the wwids file.  This also keeps the initramfs from picking
>      a
>      > different user_friendly_name than will be picked in late boot (but
>      > hopefully that issue has been sorted out by other patches already).
>      >
>      This is in fact SUSE-specific, as we do _not_ require any a-priory
>      configuration. So we will start out with a system without any wwid file,
>      _but_ expect multipath to start up properly.
>      As the wwid file will only be created by multipathd the udev check will
>      always being false, and none of the multipath devices will be created.
>      > I'm not NAKing this. The question of what to do about the differnces
>      in
>      > the distribution's systemd and udev configurations can be hashed out
>      > outside of patch reviewing. I just want to make sure I understand the
>      > race this is solving.
>      >
>      This is in fact a good question.
>      We should be coming up with a common approach which would suit both
>      needs.
> 
>      Cheers,
> 
>      Hannes
>      --
>      Dr. Hannes Reinecke                   zSeries & Storage
>      [2]hare@suse.de                          [3]+49 911 74053 688
>      SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
>      GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
> 
> References
> 
>    Visible links
>    1. mailto:hare@suse.de
>    2. mailto:hare@suse.de
>    3. file:///tmp/tel:%2B49%20911%2074053%20688

Re: [PATCH 22/57] multipath: use option '-i' when called from udev

[Hannes Reinecke]

On 05/03/2016 04:14 PM, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 08:37:08AM +0200, Christophe Varoqui wrote:
>>    Ben,
>>    does this patch actually break the Red Hat integration ?
>>    I'm inclined to merge it if not.
> 
> Like we talked about earlier, RedHat uses a completely different
> multipath.rules file, which includes some command options that were
> NAK'ed for inclusion. We can sort that out. Either we will or we won't
> come to an agreement on how multipath should interact with udev. If we
> don't we can probably just keep different rules files for the different
> distributions. But having a mostly-but-not-quite working version of the
> SUSE rules file really benefits nobody. I'm in favor of this going it.
> 
I've just send the rules file for completeness; I'm perfectly fine
with it not going in.

Or maybe having a 'multipath.rules.suse' and
'multipath.rules.redhat'; that way both rules are visible upstream
and people will know there's a difference.

Especially as we're already have RH and SUSE specific init files.
So there is a precedent.

As for systemd/udev integration itself:

For SUSE our setup is a follows:

1) We stick to the original concept of autoconfiguration, ie if
multipath/multipathd is started it will claim all devices. Any
device which should be omitted need to be blacklisted.
2) Corollary to the above: the /etc/multipath.wwids file is not
required to be present.
3) For udev integration we rely on 56-multipath.rules to set the
variable 'ENV{DM_MULTIPATH_DEVICE_PATH}="1"' if a device should be
handled by multipath; this rule also sets the 'SYSTEMD_READY=0'
variable to instruct systemd to skip this path.
This has the potential scalability issue that we need to call
'multipath' to set this variable. In our setup we call multipath
with '-u' to connect to multipathd; so if we run at a large system
the CLI interface might become overloaded.
I must admit I'm not overly happy with this design, so if anyone has
better ideas I'd be happy to accomodate it.
4) From 2) and 3) follows that we need to use the option '-i' in
56-multipath.rules to avoid failures during booting.

Seeing that there now is an option 'ignore_new_boot_devs' maybe we
can extend it to cover these differences, ie that for SUSE we will
always set this to '0', and for RedHat one would set it to '1' per
default. Alternatively we could add (yet another) configuration
variable, which could be used to switch between SUSE-style
autoconfiguration and RedHat-style wwid files.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[PATCH 23/57] multipath: remove warning 'failed to get wwid'

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipath/main.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/multipath/main.c b/multipath/main.c
index d14a913..da8fa23 100644
--- a/multipath/main.c
+++ b/multipath/main.c
@@ -284,7 +284,6 @@ configure (void)
 		int failed = get_refwwid(conf->dev, conf->dev_type, pathvec,
 					 &refwwid);
 		if (!refwwid) {
-			condlog(3, "%s: failed to get wwid", conf->dev);
 			if (failed == 2 && conf->cmd == CMD_VALID_PATH)
 				printf("%s is not a valid multipath device path\n", conf->dev);
 			else
-- 
2.6.6

Re: [PATCH 23/57] multipath: remove warning 'failed to get wwid'

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:24PM +0200, Hannes Reinecke wrote:

Again. I'd prefer leaving the messages in either at a higher log level
or with less alarming text.

-Ben

> Signed-off-by: Hannes Reinecke <hare@suse.com>
> ---
>  multipath/main.c | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/multipath/main.c b/multipath/main.c
> index d14a913..da8fa23 100644
> --- a/multipath/main.c
> +++ b/multipath/main.c
> @@ -284,7 +284,6 @@ configure (void)
>  		int failed = get_refwwid(conf->dev, conf->dev_type, pathvec,
>  					 &refwwid);
>  		if (!refwwid) {
> -			condlog(3, "%s: failed to get wwid", conf->dev);
>  			if (failed == 2 && conf->cmd == CMD_VALID_PATH)
>  				printf("%s is not a valid multipath device path\n", conf->dev);
>  			else
> -- 
> 2.6.6

Re: [PATCH 23/57] multipath: remove warning 'failed to get wwid'

[Hannes Reinecke]

On 05/02/2016 05:43 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:24PM +0200, Hannes Reinecke wrote:
> 
> Again. I'd prefer leaving the messages in either at a higher log level
> or with less alarming text.
> 
I'm fine with moving it to log level 4.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[PATCH 24/57] Add dependency on systemd-udevd.service

[Hannes Reinecke]

During startup libdevmapper checks if udev is running or not.
And if it finds udev is _not_ running it'll create the device-nodes
internally, regardless on any magic ioctl flags.
This results in the occasionally message 'conflicting device node found'
in the syslog.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/multipathd.service | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipathd/multipathd.service b/multipathd/multipathd.service
index d5f8606..eb85bee 100644
--- a/multipathd/multipathd.service
+++ b/multipathd/multipathd.service
@@ -2,7 +2,7 @@
 Description=Device-Mapper Multipath Device Controller
 Before=iscsi.service iscsid.service lvm2-activation-early.service
 Before=local-fs-pre.target
-After=multipathd.socket
+After=multipathd.socket systemd-udevd.service
 DefaultDependencies=no
 Wants=local-fs-pre.target multipathd.socket blk-availability.service
 Conflicts=shutdown.target
-- 
2.6.6

[PATCH 25/57] Load all device handler modules on startup

[Hannes Reinecke]

Upon multipathd startup the device handler modules are not
necessarily loaded, so we need to load them prior to startup.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/multipathd.service | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipathd/multipathd.service b/multipathd/multipathd.service
index eb85bee..b2afd90 100644
--- a/multipathd/multipathd.service
+++ b/multipathd/multipathd.service
@@ -11,7 +11,7 @@ Conflicts=shutdown.target
 Type=notify
 NotifyAccess=main
 LimitCORE=infinity
-ExecStartPre=/sbin/modprobe dm-multipath
+ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath
 ExecStart=/sbin/multipathd -d -s
 ExecReload=/sbin/multipathd reconfigure
 
-- 
2.6.6

[PATCH 26/57] 11-dm-mpath.rules: Only import ID_FS_XXX variables if not set

[Hannes Reinecke]

We should only import ID_FS_XXX variables if they are not set,
otherwise we might be overwriting existing results.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipath/11-dm-mpath.rules | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/multipath/11-dm-mpath.rules b/multipath/11-dm-mpath.rules
index 2e7076d..5559af3 100644
--- a/multipath/11-dm-mpath.rules
+++ b/multipath/11-dm-mpath.rules
@@ -7,11 +7,11 @@ ENV{DM_UUID}!="mpath-?*", GOTO="mpath_end"
 # We'd like to avoid this, especially within udev processing.
 ENV{DM_NR_VALID_PATHS}!="?*", IMPORT{db}="DM_NR_VALID_PATHS"
 ENV{DM_NR_VALID_PATHS}!="0", GOTO="mpath_blkid_end"
-IMPORT{db}="ID_FS_TYPE"
-IMPORT{db}="ID_FS_USAGE"
-IMPORT{db}="ID_FS_UUID"
-IMPORT{db}="ID_FS_UUID_ENC"
-IMPORT{db}="ID_FS_VERSION"
+ENV{ID_FS_TYPE}!="?*", IMPORT{db}="ID_FS_TYPE"
+ENV{ID_FS_USAGE}!="?*", IMPORT{db}="ID_FS_USAGE"
+ENV{ID_FS_UUID}!="?*", IMPORT{db}="ID_FS_UUID"
+ENV{ID_FS_ENC}!="?*", IMPORT{db}="ID_FS_UUID_ENC"
+ENV{ID_FS_VERSION}!="?*", IMPORT{db}="ID_FS_VERSION"
 LABEL="mpath_blkid_end"
 
 # Also skip all foreign rules if no path is available.
-- 
2.6.6

[PATCH 27/57] Ensure multipathd is started before systemd-udev-trigger.service

[Hannes Reinecke]

We should be starting multipathd before systemd-udev-trigger
is started to reduce the load on udev during startup.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/multipathd.service | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipathd/multipathd.service b/multipathd/multipathd.service
index b2afd90..e3d6f91 100644
--- a/multipathd/multipathd.service
+++ b/multipathd/multipathd.service
@@ -1,7 +1,7 @@
 [Unit]
 Description=Device-Mapper Multipath Device Controller
 Before=iscsi.service iscsid.service lvm2-activation-early.service
-Before=local-fs-pre.target
+Before=local-fs-pre.target systemd-udev-trigger.service
 After=multipathd.socket systemd-udevd.service
 DefaultDependencies=no
 Wants=local-fs-pre.target multipathd.socket blk-availability.service
-- 
2.6.6

[PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Hannes Reinecke]

udev since v214 is placing a shared lock on the device node
whenever it's processing the event. This introduces a race
condition with multipathd, as multipathd is processing the
event for the block device at the same time as udev is
processing the events for the partitions.
And a lock on the partitions will also be visible on the
block device itself, hence multipathd won't be able to
lock the device.
When multipath manages to take a lock on the device,
udev will fail, and consequently ignore this entire event.
Which in turn might cause the system to malfunction as it
might have been a crucial event like 'remove' or 'link down'.

So we should better use LOCK_SH here; with that the flock
call in multipathd _and_ udev will succeed and the events
can be processed.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/configure.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index 30c7259..ca20ba5 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -546,7 +546,7 @@ lock_multipath (struct multipath * mpp, int lock)
 		if (!pgp->paths)
 			continue;
 		vector_foreach_slot(pgp->paths, pp, j) {
-			if (lock && flock(pp->fd, LOCK_EX | LOCK_NB) &&
+			if (lock && flock(pp->fd, LOCK_SH | LOCK_NB) &&
 			    errno == EWOULDBLOCK)
 				goto fail;
 			else if (!lock)
-- 
2.6.6

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
> udev since v214 is placing a shared lock on the device node
> whenever it's processing the event. This introduces a race
> condition with multipathd, as multipathd is processing the
> event for the block device at the same time as udev is
> processing the events for the partitions.
> And a lock on the partitions will also be visible on the
> block device itself, hence multipathd won't be able to
> lock the device.
> When multipath manages to take a lock on the device,
> udev will fail, and consequently ignore this entire event.
> Which in turn might cause the system to malfunction as it
> might have been a crucial event like 'remove' or 'link down'.
> 
> So we should better use LOCK_SH here; with that the flock
> call in multipathd _and_ udev will succeed and the events
> can be processed.

If we switch this to a shared lock, then what's the point in having it
at all? The whole point of lock_multipath is to keep multipath and
multipathd (or two concurrent calls to multipath) from trying to create
a device at the same time, and both failing. Without an exclusive lock,
this won't stop that. We can either decide that this is an unlikely
scenario, and drop it entirely, or we can have multipath create its own
lockfiles to prevent this issue without interfering with udev. But
unless I'm missing something, this won't actually do anything.

-Ben
 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/configure.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libmultipath/configure.c b/libmultipath/configure.c
> index 30c7259..ca20ba5 100644
> --- a/libmultipath/configure.c
> +++ b/libmultipath/configure.c
> @@ -546,7 +546,7 @@ lock_multipath (struct multipath * mpp, int lock)
>  		if (!pgp->paths)
>  			continue;
>  		vector_foreach_slot(pgp->paths, pp, j) {
> -			if (lock && flock(pp->fd, LOCK_EX | LOCK_NB) &&
> +			if (lock && flock(pp->fd, LOCK_SH | LOCK_NB) &&
>  			    errno == EWOULDBLOCK)
>  				goto fail;
>  			else if (!lock)
> -- 
> 2.6.6

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Hannes Reinecke]

On 05/02/2016 06:26 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
>> udev since v214 is placing a shared lock on the device node
>> whenever it's processing the event. This introduces a race
>> condition with multipathd, as multipathd is processing the
>> event for the block device at the same time as udev is
>> processing the events for the partitions.
>> And a lock on the partitions will also be visible on the
>> block device itself, hence multipathd won't be able to
>> lock the device.
>> When multipath manages to take a lock on the device,
>> udev will fail, and consequently ignore this entire event.
>> Which in turn might cause the system to malfunction as it
>> might have been a crucial event like 'remove' or 'link down'.
>>
>> So we should better use LOCK_SH here; with that the flock
>> call in multipathd _and_ udev will succeed and the events
>> can be processed.
> 
> If we switch this to a shared lock, then what's the point in having it
> at all? The whole point of lock_multipath is to keep multipath and
> multipathd (or two concurrent calls to multipath) from trying to create
> a device at the same time, and both failing. Without an exclusive lock,
> this won't stop that. We can either decide that this is an unlikely
> scenario, and drop it entirely, or we can have multipath create its own
> lockfiles to prevent this issue without interfering with udev. But
> unless I'm missing something, this won't actually do anything.
> 
This is primarily for co-operation with udev.
As we typically do _not_ use multipath for creating device-mapper tables
the synchronisation problem between multipath and multipathd
doesn't typically occur.
At least not for us :-)

And as described above, we need this patch to co-operate with udev.
Kay was pretty adamant about _not_ changing udev here.

That said, I'm happy with dropping the lock functionality completely;
one possibility would be to use the CLI functionality to route calls
from multipath to multipathd.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Benjamin Marzinski]

On Tue, May 03, 2016 at 07:57:01AM +0200, Hannes Reinecke wrote:
> On 05/02/2016 06:26 PM, Benjamin Marzinski wrote:
> > On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
> >> udev since v214 is placing a shared lock on the device node
> >> whenever it's processing the event. This introduces a race
> >> condition with multipathd, as multipathd is processing the
> >> event for the block device at the same time as udev is
> >> processing the events for the partitions.
> >> And a lock on the partitions will also be visible on the
> >> block device itself, hence multipathd won't be able to
> >> lock the device.
> >> When multipath manages to take a lock on the device,
> >> udev will fail, and consequently ignore this entire event.
> >> Which in turn might cause the system to malfunction as it
> >> might have been a crucial event like 'remove' or 'link down'.
> >>
> >> So we should better use LOCK_SH here; with that the flock
> >> call in multipathd _and_ udev will succeed and the events
> >> can be processed.
> > 
> > If we switch this to a shared lock, then what's the point in having it
> > at all? The whole point of lock_multipath is to keep multipath and
> > multipathd (or two concurrent calls to multipath) from trying to create
> > a device at the same time, and both failing. Without an exclusive lock,
> > this won't stop that. We can either decide that this is an unlikely
> > scenario, and drop it entirely, or we can have multipath create its own
> > lockfiles to prevent this issue without interfering with udev. But
> > unless I'm missing something, this won't actually do anything.
> > 
> This is primarily for co-operation with udev.
> As we typically do _not_ use multipath for creating device-mapper tables
> the synchronisation problem between multipath and multipathd
> doesn't typically occur.
> At least not for us :-)

Well, it doesn't occur for us either, unless someone runs multipath at
EXACTLY the wrong time, at which case, I'm pretty sure the same thing
can happen in any distro. The question is, do we protect against that
very unlikely occurance.  The only time I see this actually having any
real chance to occur would be if some multipath user writes a script
that listens for new devices to be discovered, and then *helpfully* runs
multipath to update the system. I have seen this. Multiple times, in
fact. I'm not sure how much work we need to expend saving these people
from themselves, however.

> 
> And as described above, we need this patch to co-operate with udev.
> Kay was pretty adamant about _not_ changing udev here.

I'm not suggesting we ask udev to chage. If we want to keep the locking,
we need to use our own private lockfiles.

> That said, I'm happy with dropping the lock functionality completely;
> one possibility would be to use the CLI functionality to route calls
> from multipath to multipathd.

I don't see why this same issue can't happen for anyone creating any
type of dm device build out of multiple physical devices, and AFAIK, lvm
doesn't bother to protect users against this, even though it can now
autoassemble. So I'm not against dropping this either.

-Ben

> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		      zSeries & Storage
> hare@suse.de			      +49 911 74053 688
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Alasdair G Kergon]

On Tue, May 03, 2016 at 09:27:32AM -0500, Benjamin Marzinski wrote:
> I don't see why this same issue can't happen for anyone creating any
> type of dm device build out of multiple physical devices, and AFAIK, lvm
> doesn't bother to protect users against this, even though it can now
> autoassemble. So I'm not against dropping this either.
 
LVM operations are co-ordinated using a variety of locks in
/var/lock/lvm.  The prefix "A_" is used for the locks designed to avoid
concurrent activation attempts on the same volume.

Alasdair

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Benjamin Marzinski]

On Tue, May 03, 2016 at 09:27:32AM -0500, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 07:57:01AM +0200, Hannes Reinecke wrote:
> > On 05/02/2016 06:26 PM, Benjamin Marzinski wrote:
> > > On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
> > >> udev since v214 is placing a shared lock on the device node
> > >> whenever it's processing the event. This introduces a race
> > >> condition with multipathd, as multipathd is processing the
> > >> event for the block device at the same time as udev is
> > >> processing the events for the partitions.
> > >> And a lock on the partitions will also be visible on the
> > >> block device itself, hence multipathd won't be able to
> > >> lock the device.
> > >> When multipath manages to take a lock on the device,
> > >> udev will fail, and consequently ignore this entire event.
> > >> Which in turn might cause the system to malfunction as it
> > >> might have been a crucial event like 'remove' or 'link down'.
> > >>
> > >> So we should better use LOCK_SH here; with that the flock
> > >> call in multipathd _and_ udev will succeed and the events
> > >> can be processed.
> > > 
> > > If we switch this to a shared lock, then what's the point in having it
> > > at all? The whole point of lock_multipath is to keep multipath and
> > > multipathd (or two concurrent calls to multipath) from trying to create
> > > a device at the same time, and both failing. Without an exclusive lock,
> > > this won't stop that. We can either decide that this is an unlikely
> > > scenario, and drop it entirely, or we can have multipath create its own
> > > lockfiles to prevent this issue without interfering with udev. But
> > > unless I'm missing something, this won't actually do anything.
> > > 
> > This is primarily for co-operation with udev.
> > As we typically do _not_ use multipath for creating device-mapper tables
> > the synchronisation problem between multipath and multipathd
> > doesn't typically occur.
> > At least not for us :-)
> 
> Well, it doesn't occur for us either, unless someone runs multipath at
> EXACTLY the wrong time, at which case, I'm pretty sure the same thing
> can happen in any distro. The question is, do we protect against that
> very unlikely occurance.  The only time I see this actually having any
> real chance to occur would be if some multipath user writes a script
> that listens for new devices to be discovered, and then *helpfully* runs
> multipath to update the system. I have seen this. Multiple times, in
> fact. I'm not sure how much work we need to expend saving these people
> from themselves, however.
> 
> > 
> > And as described above, we need this patch to co-operate with udev.
> > Kay was pretty adamant about _not_ changing udev here.
> 
> I'm not suggesting we ask udev to chage. If we want to keep the locking,
> we need to use our own private lockfiles.
> 
> > That said, I'm happy with dropping the lock functionality completely;
> > one possibility would be to use the CLI functionality to route calls
> > from multipath to multipathd.
> 
> I don't see why this same issue can't happen for anyone creating any
> type of dm device build out of multiple physical devices, and AFAIK, lvm
> doesn't bother to protect users against this, even though it can now
> autoassemble. So I'm not against dropping this either.

Actually, after some more thought... There must be something that udev
is protecting itself against by taking the shared lock. Do you know what
that is? Assuming that multipath wants to protect against the same
thing, we probably do want to keep locking here.

So, assuming that we do want to protect ourselves against whatever udev
is protecting itself against, I remove my NAK of the original patch, and
apologize for the confusion. If we don't need to worry about udev's
issue, then we can try just removing the locks.  At worst, we'll find
out quickly what they are important for.

-Ben

> 
> -Ben
> 
> > Cheers,
> > 
> > Hannes
> > -- 
> > Dr. Hannes Reinecke		      zSeries & Storage
> > hare@suse.de			      +49 911 74053 688
> > SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
> > GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Germano Percossi]

Hi,

Sorry for jumping in the middle of patch review

On 05/03/2016 03:27 PM, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 07:57:01AM +0200, Hannes Reinecke wrote:
>> On 05/02/2016 06:26 PM, Benjamin Marzinski wrote:
>>> On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
>> This is primarily for co-operation with udev.
>> As we typically do _not_ use multipath for creating device-mapper tables
>> the synchronisation problem between multipath and multipathd
>> doesn't typically occur.
>> At least not for us :-)
>
> Well, it doesn't occur for us either, unless someone runs multipath at
> EXACTLY the wrong time, at which case, I'm pretty sure the same thing
> can happen in any distro. The question is, do we protect against that
> very unlikely occurance.  The only time I see this actually having any
> real chance to occur would be if some multipath user writes a script
> that listens for new devices to be discovered, and then *helpfully* runs
> multipath to update the system. I have seen this. Multiple times, in
> fact. I'm not sure how much work we need to expend saving these people
> from themselves, however.
>

I just want to make sure XenServer is not among those that need to
be saved from themselves.
We do mix multipathd and multipath commands but in a way that we think
is safe: we add devices with "multipath -r" and remove with "multipath -f".
This is because we slightly bend the wwids file usage to be used
as a blacklisting mechanism (in our own way).

I am wondering if removing cooperation between multipath and multipathd
is going to cause us troubles in the future.

Regards,
Germano

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Benjamin Marzinski]

On Tue, May 03, 2016 at 04:31:19PM +0100, Germano Percossi wrote:
> Hi,
> 
> Sorry for jumping in the middle of patch review
> 
> On 05/03/2016 03:27 PM, Benjamin Marzinski wrote:
> >On Tue, May 03, 2016 at 07:57:01AM +0200, Hannes Reinecke wrote:
> >>On 05/02/2016 06:26 PM, Benjamin Marzinski wrote:
> >>>On Wed, Apr 27, 2016 at 01:10:29PM +0200, Hannes Reinecke wrote:
> >>This is primarily for co-operation with udev.
> >>As we typically do _not_ use multipath for creating device-mapper tables
> >>the synchronisation problem between multipath and multipathd
> >>doesn't typically occur.
> >>At least not for us :-)
> >
> >Well, it doesn't occur for us either, unless someone runs multipath at
> >EXACTLY the wrong time, at which case, I'm pretty sure the same thing
> >can happen in any distro. The question is, do we protect against that
> >very unlikely occurance.  The only time I see this actually having any
> >real chance to occur would be if some multipath user writes a script
> >that listens for new devices to be discovered, and then *helpfully* runs
> >multipath to update the system. I have seen this. Multiple times, in
> >fact. I'm not sure how much work we need to expend saving these people
> >from themselves, however.
> >
> 
> I just want to make sure XenServer is not among those that need to
> be saved from themselves.
> We do mix multipathd and multipath commands but in a way that we think
> is safe: we add devices with "multipath -r" and remove with "multipath -f".
> This is because we slightly bend the wwids file usage to be used
> as a blacklisting mechanism (in our own way).

The only way you would be likely to run into problems is if you would be
running "multipath -r" at the same time as the udev add event for the
first path device was happening.  We only lock the devices on the
initial create.

But doesn't multipathd try to add these devices too?

-Ben
 
> I am wondering if removing cooperation between multipath and multipathd
> is going to cause us troubles in the future.
> 
> Regards,
> Germano
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: [PATCH 28/57] libmultipath: use a shared lock to co-operate with udev

[Germano Percossi]

Hi Ben,

On 05/03/2016 09:23 PM, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 04:31:19PM +0100, Germano Percossi wrote:
>>
>> I just want to make sure XenServer is not among those that need to
>> be saved from themselves.
>> We do mix multipathd and multipath commands but in a way that we think
>> is safe: we add devices with "multipath -r" and remove with "multipath -f".
>> This is because we slightly bend the wwids file usage to be used
>> as a blacklisting mechanism (in our own way).
>
> The only way you would be likely to run into problems is if you would be
> running "multipath -r" at the same time as the udev add event for the
> first path device was happening.  We only lock the devices on the
> initial create.
>
> But doesn't multipathd try to add these devices too?

In our case, no.
What we did is to remove condition 1 for "find_multipaths yes".
So, unless someone (or something) else adds the wwid to the
wwids file, the path is ignored.
Doing this way, it is our storage manager that decides when it is time 
for a path to be used by multipath and, to this end, it invokes
multipath -r.

So, from our point of view, there should not be interactions with
multipathd but I could be wrong.

For what concerns udev, I am not sure what are your intentions about
the lock.
If it is just to protect udev and multipathd, then we could have 
problems with multipath.

Germano

>
> -Ben
>

[PATCH 29/57] Only filter for udev property if uid_attribute is present

[Hannes Reinecke]

It only makes sense to filter for an existing udev property if the
'uid_attribute' variable is set. Otherwise multipath did not
get the uid from udev and so we might not even have udev information.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/configure.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index ca20ba5..0076485 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -967,7 +967,8 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)
 				return ret;
 			}
 		}
-		if (pp->udev && filter_property(conf, pp->udev) > 0)
+		if (pp->udev && pp->uid_attribute &&
+		    filter_property(conf, pp->udev) > 0)
 			return 2;
 
 		refwwid = pp->wwid;
@@ -994,7 +995,8 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)
 				return ret;
 			}
 		}
-		if (pp->udev && filter_property(conf, pp->udev) > 0)
+		if (pp->udev && pp->uid_attribute &&
+		    filter_property(conf, pp->udev) > 0)
 			return 2;
 
 		refwwid = pp->wwid;
@@ -1017,7 +1019,8 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)
 					dev);
 			return ret;
 		}
-		if (pp->udev && filter_property(conf, pp->udev) > 0)
+		if (pp->udev && pp->uid_attribute &&
+		    filter_property(conf, pp->udev) > 0)
 			return 2;
 
 		refwwid = pp->wwid;
-- 
2.6.6

[PATCH 30/57] multipathd: skip uninitialized devices during reconfiguration

[Hannes Reinecke]

libudev has a separate filter setting to skip uninitialized
devices during enumeration.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/discovery.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 2fa0907..2cf2566 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -148,6 +148,7 @@ path_discovery (vector pathvec, struct config * conf, int flag)
 		return -ENOMEM;
 
 	udev_enumerate_add_match_subsystem(udev_iter, "block");
+	udev_enumerate_add_match_is_initialized(udev_iter);
 	udev_enumerate_scan_devices(udev_iter);
 
 	udev_list_entry_foreach(entry,
-- 
2.6.6

Re: [PATCH 30/57] multipathd: skip uninitialized devices during reconfiguration

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:31PM +0200, Hannes Reinecke wrote:
> libudev has a separate filter setting to skip uninitialized
> devices during enumeration.

So, I assume if udev filters out a device here, it's because a uevent
should be coming when the device is fully initialized. correct?

The only thing I worry about with taking out all of the backup stuff is
what happens if udev simply times out processing an event, and doesn't
grab the UUID.  In this case we still won't get a UUID unless we force
another event, or grab the information ourselves.

But ACK on the patch itself.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/discovery.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
> index 2fa0907..2cf2566 100644
> --- a/libmultipath/discovery.c
> +++ b/libmultipath/discovery.c
> @@ -148,6 +148,7 @@ path_discovery (vector pathvec, struct config * conf, int flag)
>  		return -ENOMEM;
>  
>  	udev_enumerate_add_match_subsystem(udev_iter, "block");
> +	udev_enumerate_add_match_is_initialized(udev_iter);
>  	udev_enumerate_scan_devices(udev_iter);
>  
>  	udev_list_entry_foreach(entry,
> -- 
> 2.6.6

Re: [PATCH 30/57] multipathd: skip uninitialized devices during reconfiguration

[Hannes Reinecke]

On 05/02/2016 09:14 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:31PM +0200, Hannes Reinecke wrote:
>> libudev has a separate filter setting to skip uninitialized
>> devices during enumeration.
> 
> So, I assume if udev filters out a device here, it's because a uevent
> should be coming when the device is fully initialized. correct?
> 
Correct.

> The only thing I worry about with taking out all of the backup stuff is
> what happens if udev simply times out processing an event, and doesn't
> grab the UUID.  In this case we still won't get a UUID unless we force
> another event, or grab the information ourselves.
> 
Well, but we won't be seeing the event, either.
(If udev has timed out an event it won't be processed further, so it
will never show up on the udev event queue.)

However, there is a reason why I posted all work-arounds we've queued
together with this patch.

In the end, the udev integration stuff has fanned out across several
patches, where it's now really hard to figure out which bits are really
required.
So I'd prefer to leave them in; we can always remove them later.

> But ACK on the patch itself.
> 
Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[PATCH 31/57] multipathd: improve uxlsnr

[Hannes Reinecke]

Improve uxlsnr by adding a lock around the poll structure and
not reallocating memory for every poll, thereby significantly
speedup processing.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipathd/main.c   |  6 ++--
 multipathd/uxlsnr.c | 84 +++++++++++++++++++++++++++++++----------------------
 2 files changed, 53 insertions(+), 37 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 61b82f6..c1b94bd 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -966,8 +966,10 @@ uevqloop (void * ap)
 static void *
 uxlsnrloop (void * ap)
 {
-	if (cli_init())
+	if (cli_init()) {
+		condlog(1, "Failed to init uxsock listener");
 		return NULL;
+	}
 
 	set_handler_callback(LIST+PATHS, cli_list_paths);
 	set_handler_callback(LIST+PATHS+FMT, cli_list_paths_fmt);
@@ -1278,7 +1280,7 @@ check_path (struct vectors * vecs, struct path * pp)
 		sysfs_attr_set_value(pp->udev, "uevent", "change",
 				     strlen("change"));
 		return 0;
-	} 
+	}
 
 	/*
 	 * provision a next check soonest,
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index e5c5d90..77efa8a 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -44,6 +44,8 @@ struct client {
 	int fd;
 };
 
+#define MIN_POLLS 1023
+
 LIST_HEAD(clients);
 pthread_mutex_t client_lock = PTHREAD_MUTEX_INITIALIZER;
 struct pollfd *polls;
@@ -133,6 +135,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 	char *inbuf;
 	char *reply;
 	sigset_t mask;
+	int old_clients = MIN_POLLS;
 
 	ux_sock = ux_socket_listen(DEFAULT_SOCKET);
 
@@ -142,20 +145,22 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 	}
 
 	if (!conf) {
-		condlog(1, "configuration changed");
+		condlog(1, "uxsock: configuration changed");
 		return NULL;
 	}
 
-	timeout = conf->uxsock_timeout;
-
 	pthread_cleanup_push(uxsock_cleanup, NULL);
 
-	polls = (struct pollfd *)MALLOC(0);
+	condlog(3, "uxsock: startup listener");
+	polls = (struct pollfd *)MALLOC(MIN_POLLS + 1);
+	if (!polls) {
+		condlog(0, "uxsock: failed to allocate poll fds");
+		return NULL;
+	}
 	pthread_sigmask(SIG_SETMASK, NULL, &mask);
 	sigdelset(&mask, SIGHUP);
 	sigdelset(&mask, SIGUSR1);
 	while (1) {
-		struct pollfd *new;
 		struct client *c, *tmp;
 		int i, poll_count, num_clients;
 
@@ -173,19 +178,25 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 		list_for_each_entry(c, &clients, node) {
 			num_clients++;
 		}
-		new = REALLOC(polls, (1+num_clients) * sizeof(*polls));
-		/* If we can't allocate poliing space for the new client,
-		 * close it */
-		if (!new) {
-			if (!num_clients) {
-				condlog(1, "can't listen for new clients");
-				return NULL;
+		if (num_clients != old_clients) {
+			struct pollfd *new;
+			if (num_clients < MIN_POLLS) {
+				new = REALLOC(polls, (1 + MIN_POLLS) *
+						sizeof(struct pollfd));
+			} else {
+				new = REALLOC(polls, (1+num_clients) *
+						sizeof(struct pollfd));
 			}
-			dead_client(list_entry(clients.prev,
-					       typeof(struct client), node));
-		}
-		else
+			if (!new) {
+				pthread_mutex_unlock(&client_lock);
+				condlog(0, "%s: failed to realloc %d poll fds",
+					"uxsock", 1 + num_clients);
+				pthread_yield();
+				continue;
+			}
+			num_clients = old_clients;
 			polls = new;
+		}
 		polls[0].fd = ux_sock;
 		polls[0].events = POLLIN;
 
@@ -208,8 +219,8 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 			}
 
 			/* something went badly wrong! */
-			condlog(0, "poll");
-			pthread_exit(NULL);
+			condlog(0, "uxsock: poll failed with %d", errno);
+			break;
 		}
 
 		if (poll_count == 0)
@@ -230,7 +241,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 				}
 				pthread_mutex_unlock(&client_lock);
 				if (!c) {
-					condlog(3, "cli%d: invalid fd %d",
+					condlog(4, "cli%d: new fd %d",
 						i, polls[i].fd);
 					continue;
 				}
@@ -238,24 +249,27 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 					start_time.tv_sec = 0;
 				if (recv_packet(c->fd, &inbuf, timeout) != 0) {
 					dead_client(c);
-				} else {
-					condlog(4, "Got request [%s]", inbuf);
-					uxsock_trigger(inbuf, &reply, &rlen,
-						       trigger_data);
-					if (reply) {
-						if (send_packet(c->fd,
-								reply) != 0) {
-							dead_client(c);
-						}
-						condlog(4, "Reply [%d bytes]",
-							rlen);
-						FREE(reply);
-						reply = NULL;
+					continue;
+				}
+				condlog(4, "cli[%d]: Got request [%s]",
+					i, inbuf);
+				uxsock_trigger(inbuf, &reply, &rlen,
+					       trigger_data);
+				if (reply) {
+					if (send_packet(c->fd,
+							reply) != 0) {
+						dead_client(c);
+					} else {
+						condlog(4, "cli[%d]: "
+							"Reply [%d bytes]",
+							i, rlen);
 					}
-					check_timeout(start_time, inbuf,
-						      timeout);
-					FREE(inbuf);
+					FREE(reply);
+					reply = NULL;
 				}
+				check_timeout(start_time, inbuf,
+					      timeout);
+				FREE(inbuf);
 			}
 		}
 
-- 
2.6.6

[PATCH 32/57] libmultipath: remove 'needsync' argument from dm_simplecmd_noflush

[Hannes Reinecke]

Argument is always '1'.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/configure.c | 2 +-
 libmultipath/devmapper.c | 6 +++---
 libmultipath/devmapper.h | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index 0076485..d093cfd 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -634,7 +634,7 @@ domap (struct multipath * mpp, char * params)
 	case ACT_RESIZE:
 		r = dm_addmap_reload(mpp, params);
 		if (r)
-			r = dm_simplecmd_flush(DM_DEVICE_RESUME, mpp->alias, 1, 0);
+			r = dm_simplecmd_flush(DM_DEVICE_RESUME, mpp->alias, 0);
 		break;
 
 	case ACT_RENAME:
diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 36c1a20..1115216 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -248,8 +248,8 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_t
 }
 
 extern int
-dm_simplecmd_flush (int task, const char *name, int needsync, uint16_t udev_flags) {
-	return dm_simplecmd(task, name, 0, needsync, udev_flags, 0);
+dm_simplecmd_flush (int task, const char *name, uint16_t udev_flags) {
+	return dm_simplecmd(task, name, 0, 1, udev_flags, 0);
 }
 
 extern int
@@ -833,7 +833,7 @@ dm_suspend_and_flush_map (const char * mapname)
 	if (s)
 		queue_if_no_path = 0;
 	else
-		s = dm_simplecmd_flush(DM_DEVICE_SUSPEND, mapname, 1, 0);
+		s = dm_simplecmd_flush(DM_DEVICE_SUSPEND, mapname, 0);
 
 	if (!dm_flush_map(mapname)) {
 		condlog(4, "multipath map %s removed", mapname);
diff --git a/libmultipath/devmapper.h b/libmultipath/devmapper.h
index 0d27723..8dd0963 100644
--- a/libmultipath/devmapper.h
+++ b/libmultipath/devmapper.h
@@ -15,7 +15,7 @@
 void dm_init(void);
 int dm_prereq (void);
 int dm_drv_version (unsigned int * version, char * str);
-int dm_simplecmd_flush (int, const char *, int, uint16_t);
+int dm_simplecmd_flush (int, const char *, uint16_t);
 int dm_simplecmd_noflush (int, const char *, int, uint16_t);
 int dm_addmap_create (struct multipath *mpp, char *params);
 int dm_addmap_reload (struct multipath *mpp, char *params);
-- 
2.6.6

[PATCH 33/57] libmultipath: remove 'use_uuid' argument from dm_addmap()

[Hannes Reinecke]

'use_uuid' was only ever set for DM_DEVICE_CREATE, so we
can check the task and remove the 'use_uuid' argument.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 1115216..c7d4b49 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -264,8 +264,8 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
 }
 
 extern int
-dm_addmap (int task, const char *target, struct multipath *mpp, char * params,
-	   int use_uuid, int ro) {
+dm_addmap (int task, const char *target, struct multipath *mpp,
+	   char * params, int ro) {
 	int r = 0;
 	struct dm_task *dmt;
 	char *prefixed_uuid = NULL;
@@ -283,7 +283,7 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,
 	if (ro)
 		dm_task_set_ro(dmt);
 
-	if (use_uuid && strlen(mpp->wwid) > 0){
+	if ((task == DM_DEVICE_CREATE) && strlen(mpp->wwid) > 0){
 		prefixed_uuid = MALLOC(UUID_PREFIX_LEN + strlen(mpp->wwid) + 1);
 		if (!prefixed_uuid) {
 			condlog(0, "cannot create prefixed uuid : %s",
@@ -339,8 +339,7 @@ dm_addmap_create (struct multipath *mpp, char * params) {
 	for (ro = 0; ro <= 1; ro++) {
 		int err;
 
-		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
-			      mpp, params, 1, ro))
+		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
 			return 1;
 		/*
 		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
@@ -365,11 +364,11 @@ dm_addmap_create (struct multipath *mpp, char * params) {
 
 extern int
 dm_addmap_reload (struct multipath *mpp, char *params) {
-	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, 0, ADDMAP_RW))
+	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
 		return 1;
 	if (errno != EROFS)
 		return 0;
-	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, 0, ADDMAP_RO);
+	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
 }
 
 extern int
-- 
2.6.6

[PATCH 34/57] Always set DM_UDEV_DISABLE_LIBRARY_FALLBACK

[Hannes Reinecke]

The 'DM_UDEV_DISABLE_LIBRARY_FALLBACK' should be set always,
as with modern installations udev is responsible for maintaining
the '/dev/mapper' directory.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index c7d4b49..2fedf51 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -230,7 +230,9 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_t
 	if (do_deferred(deferred_remove))
 		dm_task_deferred_remove(dmt);
 #endif
-	if (udev_wait_flag && !dm_task_set_cookie(dmt, &cookie, ((conf->daemon)? DM_UDEV_DISABLE_LIBRARY_FALLBACK : 0) | udev_flags)) {
+	if (udev_wait_flag &&
+	    !dm_task_set_cookie(dmt, &cookie,
+				DM_UDEV_DISABLE_LIBRARY_FALLBACK | udev_flags)) {
 		dm_udev_complete(cookie);
 		goto out;
 	}
@@ -310,7 +312,8 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 	dm_task_no_open_count(dmt);
 
 	if (task == DM_DEVICE_CREATE &&
-	    !dm_task_set_cookie(dmt, &cookie, (conf->daemon)? DM_UDEV_DISABLE_LIBRARY_FALLBACK : 0)) {
+	    !dm_task_set_cookie(dmt, &cookie,
+				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
 		dm_udev_complete(cookie);
 		goto freeout;
 	}
@@ -1401,7 +1404,8 @@ dm_rename (const char * old, char * new)
 
 	dm_task_no_open_count(dmt);
 
-	if (!dm_task_set_cookie(dmt, &cookie, (conf->daemon)? DM_UDEV_DISABLE_LIBRARY_FALLBACK : 0))
+	if (!dm_task_set_cookie(dmt, &cookie,
+				DM_UDEV_DISABLE_LIBRARY_FALLBACK))
 		goto out;
 	r = dm_task_run(dmt);
 
-- 
2.6.6

[PATCH 35/57] libmultipath: pass in cookie as argument for dm_simplecmd()

[Hannes Reinecke]

Instead of generating the cookie internally we should be passing
it in as an argument; that will allow for cookie reuse.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 2fedf51..e6156f7 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -207,11 +207,12 @@ dm_prereq (void)
 #define do_deferred(x) ((x) == DEFERRED_REMOVE_ON || (x) == DEFERRED_REMOVE_IN_PROGRESS)
 
 static int
-dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_t udev_flags, int deferred_remove) {
+dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
+	      uint16_t udev_flags, int deferred_remove, uint32_t *cookie)
+{
 	int r = 0;
 	int udev_wait_flag = (need_sync && (task == DM_DEVICE_RESUME ||
 					    task == DM_DEVICE_REMOVE));
-	uint32_t cookie = 0;
 	struct dm_task *dmt;
 
 	if (!(dmt = dm_task_create (task)))
@@ -231,18 +232,18 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_t
 		dm_task_deferred_remove(dmt);
 #endif
 	if (udev_wait_flag &&
-	    !dm_task_set_cookie(dmt, &cookie,
+	    !dm_task_set_cookie(dmt, cookie,
 				DM_UDEV_DISABLE_LIBRARY_FALLBACK | udev_flags)) {
-		dm_udev_complete(cookie);
+		dm_udev_complete(*cookie);
 		goto out;
 	}
 	r = dm_task_run (dmt);
 
 	if (udev_wait_flag) {
 		if (!r)
-			dm_udev_complete(cookie);
+			dm_udev_complete(*cookie);
 		else
-			dm_udev_wait(cookie);
+			dm_udev_wait(*cookie);
 	}
 	out:
 	dm_task_destroy (dmt);
@@ -251,18 +252,24 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_t
 
 extern int
 dm_simplecmd_flush (int task, const char *name, uint16_t udev_flags) {
-	return dm_simplecmd(task, name, 0, 1, udev_flags, 0);
+	uint32_t cookie = 0;
+
+	return dm_simplecmd(task, name, 0, 1, udev_flags, 0, &cookie);
 }
 
 extern int
 dm_simplecmd_noflush (int task, const char *name, int needsync, uint16_t udev_flags) {
-	return dm_simplecmd(task, name, 1, needsync, udev_flags, 0);
+	uint32_t cookie = 0;
+
+	return dm_simplecmd(task, name, 1, needsync, udev_flags, 0, &cookie);
 }
 
 static int
 dm_device_remove (const char *name, int needsync, int deferred_remove) {
+	uint32_t cookie = 0;
+
 	return dm_simplecmd(DM_DEVICE_REMOVE, name, 0, needsync, 0,
-			    deferred_remove);
+			    deferred_remove, &cookie);
 }
 
 extern int
-- 
2.6.6

[PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Hannes Reinecke]

Instead of generating the cookie internally we should be
passing in the cookie to dm_addmap().

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index e6156f7..a96cc0e 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
 
 extern int
 dm_addmap (int task, const char *target, struct multipath *mpp,
-	   char * params, int ro) {
+	   char * params, int ro, uint32_t *cookie) {
 	int r = 0;
 	struct dm_task *dmt;
 	char *prefixed_uuid = NULL;
-	uint32_t cookie = 0;
 
 	if (!(dmt = dm_task_create (task)))
 		return 0;
@@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 	dm_task_no_open_count(dmt);
 
 	if (task == DM_DEVICE_CREATE &&
-	    !dm_task_set_cookie(dmt, &cookie,
+	    !dm_task_set_cookie(dmt, cookie,
 				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
-		dm_udev_complete(cookie);
+		dm_udev_complete(*cookie);
 		goto freeout;
 	}
 	r = dm_task_run (dmt);
 
 	if (task == DM_DEVICE_CREATE) {
 		if (!r)
-			dm_udev_complete(cookie);
+			dm_udev_complete(*cookie);
 		else
-			dm_udev_wait(cookie);
+			dm_udev_wait(*cookie);
 	}
 	freeout:
 	if (prefixed_uuid)
@@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 extern int
 dm_addmap_create (struct multipath *mpp, char * params) {
 	int ro;
+	uint32_t cookie = 0;
 
 	for (ro = 0; ro <= 1; ro++) {
 		int err;
 
-		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
+		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
+			      mpp, params, ro, &cookie))
 			return 1;
 		/*
 		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
@@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
 
 extern int
 dm_addmap_reload (struct multipath *mpp, char *params) {
-	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
+	uint32_t cookie = 0;
+
+	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
+		      ADDMAP_RW, &cookie))
 		return 1;
 	if (errno != EROFS)
 		return 0;
-	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
+	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
+			 ADDMAP_RO, &cookie);
 }
 
 extern int
-- 
2.6.6

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
> Instead of generating the cookie internally we should be
> passing in the cookie to dm_addmap().

These look like they could actually cause problems. With
dm_addmap_create and dm_addmap_reload, you could be in a situation where
you get a cookie back on your first call to dm_task_set_cookie, wait on
it, so that cookie is no longer in use, and then use that same cookie
id again. It's possible that after you first waited on the cookie,
someone else could have been given that cookie id (without looking into
the dm cookie code more, I'm not sure how likely this is). If that is
so, the second call to dm_addmap would be adding its command to the list
of other commands for that cookie (since you are allowed to use a cookie
for multiple dm commands). Calling dm_udev_wait would make it wait of
all the commands.

We can use a cookie multiple times and then wait on it once. But we
can't keep reusing the cookie after we've waited on it, because someone
else could be using the same cookie.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/devmapper.c | 23 ++++++++++++++---------
>  1 file changed, 14 insertions(+), 9 deletions(-)
> 
> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
> index e6156f7..a96cc0e 100644
> --- a/libmultipath/devmapper.c
> +++ b/libmultipath/devmapper.c
> @@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
>  
>  extern int
>  dm_addmap (int task, const char *target, struct multipath *mpp,
> -	   char * params, int ro) {
> +	   char * params, int ro, uint32_t *cookie) {
>  	int r = 0;
>  	struct dm_task *dmt;
>  	char *prefixed_uuid = NULL;
> -	uint32_t cookie = 0;
>  
>  	if (!(dmt = dm_task_create (task)))
>  		return 0;
> @@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>  	dm_task_no_open_count(dmt);
>  
>  	if (task == DM_DEVICE_CREATE &&
> -	    !dm_task_set_cookie(dmt, &cookie,
> +	    !dm_task_set_cookie(dmt, cookie,
>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
> -		dm_udev_complete(cookie);
> +		dm_udev_complete(*cookie);
>  		goto freeout;
>  	}
>  	r = dm_task_run (dmt);
>  
>  	if (task == DM_DEVICE_CREATE) {
>  		if (!r)
> -			dm_udev_complete(cookie);
> +			dm_udev_complete(*cookie);
>  		else
> -			dm_udev_wait(cookie);
> +			dm_udev_wait(*cookie);
>  	}
>  	freeout:
>  	if (prefixed_uuid)
> @@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>  extern int
>  dm_addmap_create (struct multipath *mpp, char * params) {
>  	int ro;
> +	uint32_t cookie = 0;
>  
>  	for (ro = 0; ro <= 1; ro++) {
>  		int err;
>  
> -		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
> +		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
> +			      mpp, params, ro, &cookie))
>  			return 1;
>  		/*
>  		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
> @@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
>  
>  extern int
>  dm_addmap_reload (struct multipath *mpp, char *params) {
> -	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
> +	uint32_t cookie = 0;
> +
> +	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> +		      ADDMAP_RW, &cookie))
>  		return 1;
>  	if (errno != EROFS)
>  		return 0;
> -	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
> +	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> +			 ADDMAP_RO, &cookie);
>  }
>  
>  extern int
> -- 
> 2.6.6

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Hannes Reinecke]

On 05/03/2016 12:23 AM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
>> Instead of generating the cookie internally we should be
>> passing in the cookie to dm_addmap().
> 
> These look like they could actually cause problems. With
> dm_addmap_create and dm_addmap_reload, you could be in a situation where
> you get a cookie back on your first call to dm_task_set_cookie, wait on
> it, so that cookie is no longer in use, and then use that same cookie
> id again. It's possible that after you first waited on the cookie,
> someone else could have been given that cookie id (without looking into
> the dm cookie code more, I'm not sure how likely this is). If that is
> so, the second call to dm_addmap would be adding its command to the list
> of other commands for that cookie (since you are allowed to use a cookie
> for multiple dm commands). Calling dm_udev_wait would make it wait of
> all the commands.
> 
> We can use a cookie multiple times and then wait on it once. But we
> can't keep reusing the cookie after we've waited on it, because someone
> else could be using the same cookie.
> 
Ah. Thanks for the clarification. I'll be looking into it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Hannes Reinecke]

On 05/03/2016 12:23 AM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
>> Instead of generating the cookie internally we should be
>> passing in the cookie to dm_addmap().
> 
> These look like they could actually cause problems. With
> dm_addmap_create and dm_addmap_reload, you could be in a situation where
> you get a cookie back on your first call to dm_task_set_cookie, wait on
> it, so that cookie is no longer in use, and then use that same cookie
> id again. It's possible that after you first waited on the cookie,
> someone else could have been given that cookie id (without looking into
> the dm cookie code more, I'm not sure how likely this is). If that is
> so, the second call to dm_addmap would be adding its command to the list
> of other commands for that cookie (since you are allowed to use a cookie
> for multiple dm commands). Calling dm_udev_wait would make it wait of
> all the commands.
> 
> We can use a cookie multiple times and then wait on it once. But we
> can't keep reusing the cookie after we've waited on it, because someone
> else could be using the same cookie.
> 
> -Ben
> 
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  libmultipath/devmapper.c | 23 ++++++++++++++---------
>>  1 file changed, 14 insertions(+), 9 deletions(-)
>>
>> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
>> index e6156f7..a96cc0e 100644
>> --- a/libmultipath/devmapper.c
>> +++ b/libmultipath/devmapper.c
>> @@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
>>  
>>  extern int
>>  dm_addmap (int task, const char *target, struct multipath *mpp,
>> -	   char * params, int ro) {
>> +	   char * params, int ro, uint32_t *cookie) {
>>  	int r = 0;
>>  	struct dm_task *dmt;
>>  	char *prefixed_uuid = NULL;
>> -	uint32_t cookie = 0;
>>  
>>  	if (!(dmt = dm_task_create (task)))
>>  		return 0;
>> @@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>>  	dm_task_no_open_count(dmt);
>>  
>>  	if (task == DM_DEVICE_CREATE &&
>> -	    !dm_task_set_cookie(dmt, &cookie,
>> +	    !dm_task_set_cookie(dmt, cookie,
>>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
>> -		dm_udev_complete(cookie);
>> +		dm_udev_complete(*cookie);
>>  		goto freeout;
>>  	}
>>  	r = dm_task_run (dmt);
>>  
>>  	if (task == DM_DEVICE_CREATE) {
>>  		if (!r)
>> -			dm_udev_complete(cookie);
>> +			dm_udev_complete(*cookie);
>>  		else
>> -			dm_udev_wait(cookie);
>> +			dm_udev_wait(*cookie);
>>  	}
>>  	freeout:
>>  	if (prefixed_uuid)
>> @@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>>  extern int
>>  dm_addmap_create (struct multipath *mpp, char * params) {
>>  	int ro;
>> +	uint32_t cookie = 0;
>>  
>>  	for (ro = 0; ro <= 1; ro++) {
>>  		int err;
>>  
>> -		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
>> +		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
>> +			      mpp, params, ro, &cookie))
>>  			return 1;
>>  		/*
>>  		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
>> @@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
>>  
>>  extern int
>>  dm_addmap_reload (struct multipath *mpp, char *params) {
>> -	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
>> +	uint32_t cookie = 0;
>> +
>> +	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
>> +		      ADDMAP_RW, &cookie))
>>  		return 1;
>>  	if (errno != EROFS)
>>  		return 0;
>> -	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
>> +	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
>> +			 ADDMAP_RO, &cookie);
>>  }
>>  
>>  extern int
>> -- 
>> 2.6.6
Looking closer, this entire section looks fishy.
From the lvm source code I couldn't figure out _how_ we could get
into a situation where dm_task_run() would return 0 (ie no error
occurred), but errno would be set to EROFS.

_If_ we could rely on dm_task_run() to return an error here this
whole issue wouldn't occur, as we haven't actually waited on the
cookie, and the second dm_addmap() call would be legit, even with
the same cookie.

Checking with git how the above came about.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Benjamin Marzinski]

On Tue, May 03, 2016 at 11:31:19AM +0200, Hannes Reinecke wrote:
> On 05/03/2016 12:23 AM, Benjamin Marzinski wrote:
> > On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
> >> Instead of generating the cookie internally we should be
> >> passing in the cookie to dm_addmap().
> > 
> > These look like they could actually cause problems. With
> > dm_addmap_create and dm_addmap_reload, you could be in a situation where
> > you get a cookie back on your first call to dm_task_set_cookie, wait on
> > it, so that cookie is no longer in use, and then use that same cookie
> > id again. It's possible that after you first waited on the cookie,
> > someone else could have been given that cookie id (without looking into
> > the dm cookie code more, I'm not sure how likely this is). If that is
> > so, the second call to dm_addmap would be adding its command to the list
> > of other commands for that cookie (since you are allowed to use a cookie
> > for multiple dm commands). Calling dm_udev_wait would make it wait of
> > all the commands.
> > 
> > We can use a cookie multiple times and then wait on it once. But we
> > can't keep reusing the cookie after we've waited on it, because someone
> > else could be using the same cookie.
> > 
> > -Ben
> > 
> >>
> >> Signed-off-by: Hannes Reinecke <hare@suse.de>
> >> ---
> >>  libmultipath/devmapper.c | 23 ++++++++++++++---------
> >>  1 file changed, 14 insertions(+), 9 deletions(-)
> >>
> >> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
> >> index e6156f7..a96cc0e 100644
> >> --- a/libmultipath/devmapper.c
> >> +++ b/libmultipath/devmapper.c
> >> @@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
> >>  
> >>  extern int
> >>  dm_addmap (int task, const char *target, struct multipath *mpp,
> >> -	   char * params, int ro) {
> >> +	   char * params, int ro, uint32_t *cookie) {
> >>  	int r = 0;
> >>  	struct dm_task *dmt;
> >>  	char *prefixed_uuid = NULL;
> >> -	uint32_t cookie = 0;
> >>  
> >>  	if (!(dmt = dm_task_create (task)))
> >>  		return 0;
> >> @@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
> >>  	dm_task_no_open_count(dmt);
> >>  
> >>  	if (task == DM_DEVICE_CREATE &&
> >> -	    !dm_task_set_cookie(dmt, &cookie,
> >> +	    !dm_task_set_cookie(dmt, cookie,
> >>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
> >> -		dm_udev_complete(cookie);
> >> +		dm_udev_complete(*cookie);
> >>  		goto freeout;
> >>  	}
> >>  	r = dm_task_run (dmt);
> >>  
> >>  	if (task == DM_DEVICE_CREATE) {
> >>  		if (!r)
> >> -			dm_udev_complete(cookie);
> >> +			dm_udev_complete(*cookie);
> >>  		else
> >> -			dm_udev_wait(cookie);
> >> +			dm_udev_wait(*cookie);
> >>  	}
> >>  	freeout:
> >>  	if (prefixed_uuid)
> >> @@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
> >>  extern int
> >>  dm_addmap_create (struct multipath *mpp, char * params) {
> >>  	int ro;
> >> +	uint32_t cookie = 0;
> >>  
> >>  	for (ro = 0; ro <= 1; ro++) {
> >>  		int err;
> >>  
> >> -		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
> >> +		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
> >> +			      mpp, params, ro, &cookie))
> >>  			return 1;
> >>  		/*
> >>  		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
> >> @@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
> >>  
> >>  extern int
> >>  dm_addmap_reload (struct multipath *mpp, char *params) {
> >> -	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
> >> +	uint32_t cookie = 0;
> >> +
> >> +	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> >> +		      ADDMAP_RW, &cookie))
> >>  		return 1;
> >>  	if (errno != EROFS)
> >>  		return 0;
> >> -	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
> >> +	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> >> +			 ADDMAP_RO, &cookie);
> >>  }
> >>  
> >>  extern int
> >> -- 
> >> 2.6.6
> Looking closer, this entire section looks fishy.
> >From the lvm source code I couldn't figure out _how_ we could get
> into a situation where dm_task_run() would return 0 (ie no error
> occurred), but errno would be set to EROFS.

dm_task_run returns a 0 on error, not success.

-Ben

> 
> _If_ we could rely on dm_task_run() to return an error here this
> whole issue wouldn't occur, as we haven't actually waited on the
> cookie, and the second dm_addmap() call would be legit, even with
> the same cookie.
> 
> Checking with git how the above came about.
> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		   Teamlead Storage & Networking
> hare@suse.de			               +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Hannes Reinecke]

On 05/03/2016 04:39 PM, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 11:31:19AM +0200, Hannes Reinecke wrote:
>> On 05/03/2016 12:23 AM, Benjamin Marzinski wrote:
>>> On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
>>>> Instead of generating the cookie internally we should be
>>>> passing in the cookie to dm_addmap().
>>>
>>> These look like they could actually cause problems. With
>>> dm_addmap_create and dm_addmap_reload, you could be in a situation where
>>> you get a cookie back on your first call to dm_task_set_cookie, wait on
>>> it, so that cookie is no longer in use, and then use that same cookie
>>> id again. It's possible that after you first waited on the cookie,
>>> someone else could have been given that cookie id (without looking into
>>> the dm cookie code more, I'm not sure how likely this is). If that is
>>> so, the second call to dm_addmap would be adding its command to the list
>>> of other commands for that cookie (since you are allowed to use a cookie
>>> for multiple dm commands). Calling dm_udev_wait would make it wait of
>>> all the commands.
>>>
>>> We can use a cookie multiple times and then wait on it once. But we
>>> can't keep reusing the cookie after we've waited on it, because someone
>>> else could be using the same cookie.
>>>
>>> -Ben
>>>
>>>>
>>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>>> ---
>>>>  libmultipath/devmapper.c | 23 ++++++++++++++---------
>>>>  1 file changed, 14 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
>>>> index e6156f7..a96cc0e 100644
>>>> --- a/libmultipath/devmapper.c
>>>> +++ b/libmultipath/devmapper.c
>>>> @@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
>>>>  
>>>>  extern int
>>>>  dm_addmap (int task, const char *target, struct multipath *mpp,
>>>> -	   char * params, int ro) {
>>>> +	   char * params, int ro, uint32_t *cookie) {
>>>>  	int r = 0;
>>>>  	struct dm_task *dmt;
>>>>  	char *prefixed_uuid = NULL;
>>>> -	uint32_t cookie = 0;
>>>>  
>>>>  	if (!(dmt = dm_task_create (task)))
>>>>  		return 0;
>>>> @@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>>>>  	dm_task_no_open_count(dmt);
>>>>  
>>>>  	if (task == DM_DEVICE_CREATE &&
>>>> -	    !dm_task_set_cookie(dmt, &cookie,
>>>> +	    !dm_task_set_cookie(dmt, cookie,
>>>>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
>>>> -		dm_udev_complete(cookie);
>>>> +		dm_udev_complete(*cookie);
>>>>  		goto freeout;
>>>>  	}
>>>>  	r = dm_task_run (dmt);
>>>>  
>>>>  	if (task == DM_DEVICE_CREATE) {
>>>>  		if (!r)
>>>> -			dm_udev_complete(cookie);
>>>> +			dm_udev_complete(*cookie);
>>>>  		else
>>>> -			dm_udev_wait(cookie);
>>>> +			dm_udev_wait(*cookie);
>>>>  	}
>>>>  	freeout:
>>>>  	if (prefixed_uuid)
>>>> @@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
>>>>  extern int
>>>>  dm_addmap_create (struct multipath *mpp, char * params) {
>>>>  	int ro;
>>>> +	uint32_t cookie = 0;
>>>>  
>>>>  	for (ro = 0; ro <= 1; ro++) {
>>>>  		int err;
>>>>  
>>>> -		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
>>>> +		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
>>>> +			      mpp, params, ro, &cookie))
>>>>  			return 1;
>>>>  		/*
>>>>  		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
>>>> @@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
>>>>  
>>>>  extern int
>>>>  dm_addmap_reload (struct multipath *mpp, char *params) {
>>>> -	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
>>>> +	uint32_t cookie = 0;
>>>> +
>>>> +	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
>>>> +		      ADDMAP_RW, &cookie))
>>>>  		return 1;
>>>>  	if (errno != EROFS)
>>>>  		return 0;
>>>> -	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
>>>> +	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
>>>> +			 ADDMAP_RO, &cookie);
>>>>  }
>>>>  
>>>>  extern int
>>>> -- 
>>>> 2.6.6
>> Looking closer, this entire section looks fishy.
>> >From the lvm source code I couldn't figure out _how_ we could get
>> into a situation where dm_task_run() would return 0 (ie no error
>> occurred), but errno would be set to EROFS.
> 
> dm_task_run returns a 0 on error, not success.
> 
Yeah, after several hours of debugging I've finally fixed things up.
Main reason why it worked was that multipathd disabled udev sync, so
all cookies were disabled and it doesn't really matter if and how
you called udevcomplete and stuff.

So, question:

Should be enable udev sync via

	dm_udev_set_sync_support(1);

in multipathd or not?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 36/57] libmultipath: pass in 'cookie' as argument for dm_addmap()

[Benjamin Marzinski]

On Tue, May 03, 2016 at 04:43:01PM +0200, Hannes Reinecke wrote:
> On 05/03/2016 04:39 PM, Benjamin Marzinski wrote:
> > On Tue, May 03, 2016 at 11:31:19AM +0200, Hannes Reinecke wrote:
> >> On 05/03/2016 12:23 AM, Benjamin Marzinski wrote:
> >>> On Wed, Apr 27, 2016 at 01:10:37PM +0200, Hannes Reinecke wrote:
> >>>> Instead of generating the cookie internally we should be
> >>>> passing in the cookie to dm_addmap().
> >>>
> >>> These look like they could actually cause problems. With
> >>> dm_addmap_create and dm_addmap_reload, you could be in a situation where
> >>> you get a cookie back on your first call to dm_task_set_cookie, wait on
> >>> it, so that cookie is no longer in use, and then use that same cookie
> >>> id again. It's possible that after you first waited on the cookie,
> >>> someone else could have been given that cookie id (without looking into
> >>> the dm cookie code more, I'm not sure how likely this is). If that is
> >>> so, the second call to dm_addmap would be adding its command to the list
> >>> of other commands for that cookie (since you are allowed to use a cookie
> >>> for multiple dm commands). Calling dm_udev_wait would make it wait of
> >>> all the commands.
> >>>
> >>> We can use a cookie multiple times and then wait on it once. But we
> >>> can't keep reusing the cookie after we've waited on it, because someone
> >>> else could be using the same cookie.
> >>>
> >>> -Ben
> >>>
> >>>>
> >>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
> >>>> ---
> >>>>  libmultipath/devmapper.c | 23 ++++++++++++++---------
> >>>>  1 file changed, 14 insertions(+), 9 deletions(-)
> >>>>
> >>>> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
> >>>> index e6156f7..a96cc0e 100644
> >>>> --- a/libmultipath/devmapper.c
> >>>> +++ b/libmultipath/devmapper.c
> >>>> @@ -274,11 +274,10 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
> >>>>  
> >>>>  extern int
> >>>>  dm_addmap (int task, const char *target, struct multipath *mpp,
> >>>> -	   char * params, int ro) {
> >>>> +	   char * params, int ro, uint32_t *cookie) {
> >>>>  	int r = 0;
> >>>>  	struct dm_task *dmt;
> >>>>  	char *prefixed_uuid = NULL;
> >>>> -	uint32_t cookie = 0;
> >>>>  
> >>>>  	if (!(dmt = dm_task_create (task)))
> >>>>  		return 0;
> >>>> @@ -319,18 +318,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
> >>>>  	dm_task_no_open_count(dmt);
> >>>>  
> >>>>  	if (task == DM_DEVICE_CREATE &&
> >>>> -	    !dm_task_set_cookie(dmt, &cookie,
> >>>> +	    !dm_task_set_cookie(dmt, cookie,
> >>>>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
> >>>> -		dm_udev_complete(cookie);
> >>>> +		dm_udev_complete(*cookie);
> >>>>  		goto freeout;
> >>>>  	}
> >>>>  	r = dm_task_run (dmt);
> >>>>  
> >>>>  	if (task == DM_DEVICE_CREATE) {
> >>>>  		if (!r)
> >>>> -			dm_udev_complete(cookie);
> >>>> +			dm_udev_complete(*cookie);
> >>>>  		else
> >>>> -			dm_udev_wait(cookie);
> >>>> +			dm_udev_wait(*cookie);
> >>>>  	}
> >>>>  	freeout:
> >>>>  	if (prefixed_uuid)
> >>>> @@ -345,11 +344,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
> >>>>  extern int
> >>>>  dm_addmap_create (struct multipath *mpp, char * params) {
> >>>>  	int ro;
> >>>> +	uint32_t cookie = 0;
> >>>>  
> >>>>  	for (ro = 0; ro <= 1; ro++) {
> >>>>  		int err;
> >>>>  
> >>>> -		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH, mpp, params, ro))
> >>>> +		if (dm_addmap(DM_DEVICE_CREATE, TGT_MPATH,
> >>>> +			      mpp, params, ro, &cookie))
> >>>>  			return 1;
> >>>>  		/*
> >>>>  		 * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD.
> >>>> @@ -374,11 +375,15 @@ dm_addmap_create (struct multipath *mpp, char * params) {
> >>>>  
> >>>>  extern int
> >>>>  dm_addmap_reload (struct multipath *mpp, char *params) {
> >>>> -	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RW))
> >>>> +	uint32_t cookie = 0;
> >>>> +
> >>>> +	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> >>>> +		      ADDMAP_RW, &cookie))
> >>>>  		return 1;
> >>>>  	if (errno != EROFS)
> >>>>  		return 0;
> >>>> -	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params, ADDMAP_RO);
> >>>> +	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
> >>>> +			 ADDMAP_RO, &cookie);
> >>>>  }
> >>>>  
> >>>>  extern int
> >>>> -- 
> >>>> 2.6.6
> >> Looking closer, this entire section looks fishy.
> >> >From the lvm source code I couldn't figure out _how_ we could get
> >> into a situation where dm_task_run() would return 0 (ie no error
> >> occurred), but errno would be set to EROFS.
> > 
> > dm_task_run returns a 0 on error, not success.
> > 
> Yeah, after several hours of debugging I've finally fixed things up.
> Main reason why it worked was that multipathd disabled udev sync, so
> all cookies were disabled and it doesn't really matter if and how
> you called udevcomplete and stuff.
> 
> So, question:
> 
> Should be enable udev sync via
> 
> 	dm_udev_set_sync_support(1);
> 
> in multipathd or not?

Not unless you want to block every multipathd thread that grabs the vecs
lock after every sync-able dm command, until udev processes that event.
In short, No.

I looked at multiple ideas to deal with syncing, including a seperate
waiting thread.  All of them had to potential to cause crippling
slowdowns. Then I realized that we don't need them. Multipathd itself
already listens to the udev events.  If we want to know when udev has
finished processing the uevent, we just wait for multipathd to receive
it.

That's the crux of my solution to keeping multipath from reloading a
device before udev has finished initializing it (see commit
b833425029b59106e4e6de26470990ee39d614f5, multipathd: delay reloads
during creation). We know when udev has finished setting up the device,
because it sends us a uevent afterwards.

-Ben

> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		   Teamlead Storage & Networking
> hare@suse.de			               +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

[PATCH 37/57] Remove 'udev_sync' argument from dm_simplecmd()

[Hannes Reinecke]

The 'udev_sync' attribute is pointless without a cookie, so we
can as well use the existence of the 'cookie' argument for
the same function.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/devmapper.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index a96cc0e..12ccf1a 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -207,12 +207,10 @@ dm_prereq (void)
 #define do_deferred(x) ((x) == DEFERRED_REMOVE_ON || (x) == DEFERRED_REMOVE_IN_PROGRESS)
 
 static int
-dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
+dm_simplecmd (int task, const char *name, int no_flush,
 	      uint16_t udev_flags, int deferred_remove, uint32_t *cookie)
 {
 	int r = 0;
-	int udev_wait_flag = (need_sync && (task == DM_DEVICE_RESUME ||
-					    task == DM_DEVICE_REMOVE));
 	struct dm_task *dmt;
 
 	if (!(dmt = dm_task_create (task)))
@@ -231,7 +229,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
 	if (do_deferred(deferred_remove))
 		dm_task_deferred_remove(dmt);
 #endif
-	if (udev_wait_flag &&
+	if (cookie &&
 	    !dm_task_set_cookie(dmt, cookie,
 				DM_UDEV_DISABLE_LIBRARY_FALLBACK | udev_flags)) {
 		dm_udev_complete(*cookie);
@@ -239,7 +237,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
 	}
 	r = dm_task_run (dmt);
 
-	if (udev_wait_flag) {
+	if (cookie) {
 		if (!r)
 			dm_udev_complete(*cookie);
 		else
@@ -254,22 +252,27 @@ extern int
 dm_simplecmd_flush (int task, const char *name, uint16_t udev_flags) {
 	uint32_t cookie = 0;
 
-	return dm_simplecmd(task, name, 0, 1, udev_flags, 0, &cookie);
+	if (task == DM_DEVICE_RESUME ||
+	    task == DM_DEVICE_REMOVE)
+		return dm_simplecmd(task, name, 0, udev_flags, 0, &cookie);
+	else
+		return dm_simplecmd(task, name, 0, udev_flags, 0, NULL);
 }
 
 extern int
 dm_simplecmd_noflush (int task, const char *name, int needsync, uint16_t udev_flags) {
 	uint32_t cookie = 0;
 
-	return dm_simplecmd(task, name, 1, needsync, udev_flags, 0, &cookie);
+	return dm_simplecmd(task, name, 1, udev_flags, 0,
+			    needsync ? &cookie : NULL);
 }
 
 static int
 dm_device_remove (const char *name, int needsync, int deferred_remove) {
 	uint32_t cookie = 0;
 
-	return dm_simplecmd(DM_DEVICE_REMOVE, name, 0, needsync, 0,
-			    deferred_remove, &cookie);
+	return dm_simplecmd(DM_DEVICE_REMOVE, name, 0, 0,
+			    deferred_remove, needsync ? &cookie : NULL);
 }
 
 extern int
-- 
2.6.6

Re: [PATCH 37/57] Remove 'udev_sync' argument from dm_simplecmd()

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:38PM +0200, Hannes Reinecke wrote:
> The 'udev_sync' attribute is pointless without a cookie, so we
> can as well use the existence of the 'cookie' argument for
> the same function.

ACK. But I do worry that dm_simplecmd is still going to confuse someone
later. Right now, we have a udev_flags variable that gets set seperate
from the cookie, but if there is no cookie pointer, we never set these
flags (because they are tacked on to the cookie in dm_task_set_cookie).
Right now, we never set udev_flags and not cookie.  But it's not obvious
that you shouldn't. 

-Ben
> 
> Signed-off-by: Hannes Reinecke <hare@suse.com>
> ---
>  libmultipath/devmapper.c | 21 ++++++++++++---------
>  1 file changed, 12 insertions(+), 9 deletions(-)
> 
> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
> index a96cc0e..12ccf1a 100644
> --- a/libmultipath/devmapper.c
> +++ b/libmultipath/devmapper.c
> @@ -207,12 +207,10 @@ dm_prereq (void)
>  #define do_deferred(x) ((x) == DEFERRED_REMOVE_ON || (x) == DEFERRED_REMOVE_IN_PROGRESS)
>  
>  static int
> -dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
> +dm_simplecmd (int task, const char *name, int no_flush,
>  	      uint16_t udev_flags, int deferred_remove, uint32_t *cookie)
>  {
>  	int r = 0;
> -	int udev_wait_flag = (need_sync && (task == DM_DEVICE_RESUME ||
> -					    task == DM_DEVICE_REMOVE));
>  	struct dm_task *dmt;
>  
>  	if (!(dmt = dm_task_create (task)))
> @@ -231,7 +229,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
>  	if (do_deferred(deferred_remove))
>  		dm_task_deferred_remove(dmt);
>  #endif
> -	if (udev_wait_flag &&
> +	if (cookie &&
>  	    !dm_task_set_cookie(dmt, cookie,
>  				DM_UDEV_DISABLE_LIBRARY_FALLBACK | udev_flags)) {
>  		dm_udev_complete(*cookie);
> @@ -239,7 +237,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync,
>  	}
>  	r = dm_task_run (dmt);
>  
> -	if (udev_wait_flag) {
> +	if (cookie) {
>  		if (!r)
>  			dm_udev_complete(*cookie);
>  		else
> @@ -254,22 +252,27 @@ extern int
>  dm_simplecmd_flush (int task, const char *name, uint16_t udev_flags) {
>  	uint32_t cookie = 0;
>  
> -	return dm_simplecmd(task, name, 0, 1, udev_flags, 0, &cookie);
> +	if (task == DM_DEVICE_RESUME ||
> +	    task == DM_DEVICE_REMOVE)
> +		return dm_simplecmd(task, name, 0, udev_flags, 0, &cookie);
> +	else
> +		return dm_simplecmd(task, name, 0, udev_flags, 0, NULL);
>  }
>  
>  extern int
>  dm_simplecmd_noflush (int task, const char *name, int needsync, uint16_t udev_flags) {
>  	uint32_t cookie = 0;
>  
> -	return dm_simplecmd(task, name, 1, needsync, udev_flags, 0, &cookie);
> +	return dm_simplecmd(task, name, 1, udev_flags, 0,
> +			    needsync ? &cookie : NULL);
>  }
>  
>  static int
>  dm_device_remove (const char *name, int needsync, int deferred_remove) {
>  	uint32_t cookie = 0;
>  
> -	return dm_simplecmd(DM_DEVICE_REMOVE, name, 0, needsync, 0,
> -			    deferred_remove, &cookie);
> +	return dm_simplecmd(DM_DEVICE_REMOVE, name, 0, 0,
> +			    deferred_remove, needsync ? &cookie : NULL);
>  }
>  
>  extern int
> -- 
> 2.6.6

[PATCH 38/57] libmultipath: Fixup 'DM_DEVICE_RELOAD' handling

[Hannes Reinecke]

libdevmapper has the 'quirk' that DM_DEVICE_CREATE is translated
internally into a create/load/resume sequence, and the associated
cookie will wait for the last 'resume' to complete.
However, DM_DEVICE_RELOAD has no such translation, so if there
is a cookie assigned to it the caller _cannot_ wait for it,
as the cookie will only ever be completed upon the next
DM_DEVICE_RESUME.
multipathd already has some provisions for that (but even there
the cookie handling is dodgy), but 'multipath -r' doesn't know
about this.
So to avoid any future irritations this patch updates the
dm_addmad_reload() call to handle the cookie correctly,
and removes the special handling from multipathd.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/configure.c | 16 ++++------------
 libmultipath/devmapper.c | 39 +++++++++++++++++++++++++++------------
 libmultipath/devmapper.h |  2 +-
 3 files changed, 32 insertions(+), 25 deletions(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index d093cfd..dee749b 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -625,16 +625,11 @@ domap (struct multipath * mpp, char * params)
 		break;
 
 	case ACT_RELOAD:
-		r = dm_addmap_reload(mpp, params);
-		if (r)
-			r = dm_simplecmd_noflush(DM_DEVICE_RESUME, mpp->alias,
-						 0, MPATH_UDEV_RELOAD_FLAG);
+		r = dm_addmap_reload(mpp, params, 0);
 		break;
 
 	case ACT_RESIZE:
-		r = dm_addmap_reload(mpp, params);
-		if (r)
-			r = dm_simplecmd_flush(DM_DEVICE_RESUME, mpp->alias, 0);
+		r = dm_addmap_reload(mpp, params, 1);
 		break;
 
 	case ACT_RENAME:
@@ -643,11 +638,8 @@ domap (struct multipath * mpp, char * params)
 
 	case ACT_RENAME2:
 		r = dm_rename(mpp->alias_old, mpp->alias);
-		if (r) {
-			r = dm_addmap_reload(mpp, params);
-			if (r)
-				r = dm_simplecmd_noflush(DM_DEVICE_RESUME, mpp->alias, 0, MPATH_UDEV_RELOAD_FLAG);
-		}
+		if (r)
+			r = dm_addmap_reload(mpp, params, 0);
 		break;
 
 	default:
diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 12ccf1a..15c91c9 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -315,12 +315,13 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 	if (mpp->attribute_flags & (1 << ATTR_GID) &&
 	    !dm_task_set_gid(dmt, mpp->gid))
 		goto freeout;
-	condlog(4, "%s: addmap [0 %llu %s %s]", mpp->alias, mpp->size,
+	condlog(4, "%s: %s [0 %llu %s %s]", mpp->alias,
+		task == DM_DEVICE_RELOAD ? "reload" : "addmap", mpp->size,
 		target, params);
 
 	dm_task_no_open_count(dmt);
 
-	if (task == DM_DEVICE_CREATE &&
+	if (cookie &&
 	    !dm_task_set_cookie(dmt, cookie,
 				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
 		dm_udev_complete(*cookie);
@@ -328,10 +329,11 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 	}
 	r = dm_task_run (dmt);
 
-	if (task == DM_DEVICE_CREATE) {
+	if (cookie) {
 		if (!r)
 			dm_udev_complete(*cookie);
-		else
+		else if (task != DM_DEVICE_CREATE)
+			/* Do not wait on a cookie for DM_DEVICE_RELOAD */
 			dm_udev_wait(*cookie);
 	}
 	freeout:
@@ -377,16 +379,29 @@ dm_addmap_create (struct multipath *mpp, char * params) {
 #define ADDMAP_RO 1
 
 extern int
-dm_addmap_reload (struct multipath *mpp, char *params) {
+dm_addmap_reload (struct multipath *mpp, char *params, int flush) {
+	int r;
 	uint32_t cookie = 0;
+	uint16_t udev_flags = flush ? 0 : MPATH_UDEV_RELOAD_FLAG;
 
-	if (dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
-		      ADDMAP_RW, &cookie))
-		return 1;
-	if (errno != EROFS)
-		return 0;
-	return dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
-			 ADDMAP_RO, &cookie);
+	/*
+	 * DM_DEVICE_RELOAD cannot wait on a cookie, as
+	 * the cookie will only ever be released after an
+	 * DM_DEVICE_RESUME. So call DM_DEVICE_RESUME
+	 * after each DM_DEVICE_RELOAD.
+	 */
+	r = dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
+		      ADDMAP_RW, &cookie);
+	if (!r) {
+		if (errno != EROFS)
+			return 0;
+		r = dm_addmap(DM_DEVICE_RELOAD, TGT_MPATH, mpp, params,
+			      ADDMAP_RO, &cookie);
+	}
+	if (r)
+		r = dm_simplecmd(DM_DEVICE_RESUME, mpp->alias, flush,
+				 udev_flags, 0, &cookie);
+	return r;
 }
 
 extern int
diff --git a/libmultipath/devmapper.h b/libmultipath/devmapper.h
index 8dd0963..97f3742 100644
--- a/libmultipath/devmapper.h
+++ b/libmultipath/devmapper.h
@@ -18,7 +18,7 @@ int dm_drv_version (unsigned int * version, char * str);
 int dm_simplecmd_flush (int, const char *, uint16_t);
 int dm_simplecmd_noflush (int, const char *, int, uint16_t);
 int dm_addmap_create (struct multipath *mpp, char *params);
-int dm_addmap_reload (struct multipath *mpp, char *params);
+int dm_addmap_reload (struct multipath *mpp, char *params, int flush);
 int dm_map_present (const char *);
 int dm_get_map(const char *, unsigned long long *, char *);
 int dm_get_status(char *, char *);
-- 
2.6.6

[PATCH 39/57] devmapper: do not flush I/O for DM_DEVICE_CREATE

[Hannes Reinecke]

DM_DEVICE_CREATE loads a new table, so there cannot be any
I/O pending. Hence we should be setting the 'no flush'
and 'skip lockfs' flag to avoid delays during creation.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 15c91c9..c2ae83b 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -321,11 +321,16 @@ dm_addmap (int task, const char *target, struct multipath *mpp,
 
 	dm_task_no_open_count(dmt);
 
-	if (cookie &&
-	    !dm_task_set_cookie(dmt, cookie,
-				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
-		dm_udev_complete(*cookie);
-		goto freeout;
+	if (cookie) {
+		if (!dm_task_set_cookie(dmt, cookie,
+					DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
+			dm_udev_complete(*cookie);
+			goto freeout;
+		}
+		dm_task_skip_lockfs(dmt);
+#ifdef LIBDM_API_FLUSH
+		dm_task_no_flush(dmt);
+#endif
 	}
 	r = dm_task_run (dmt);
 
-- 
2.6.6

[PATCH 40/57] libmultipath: fixup dm_rename to complete cookie on failure

[Hannes Reinecke]

From my understanding we should be calling udev_complete() on
a cookie if dm_task_set_cookie() failed.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/devmapper.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index c2ae83b..b10f9e6 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -1440,8 +1440,10 @@ dm_rename (const char * old, char * new)
 	dm_task_no_open_count(dmt);
 
 	if (!dm_task_set_cookie(dmt, &cookie,
-				DM_UDEV_DISABLE_LIBRARY_FALLBACK))
+				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
+		dm_udev_complete(cookie);
 		goto out;
+	}
 	r = dm_task_run(dmt);
 
 	if (!r)
-- 
2.6.6

Re: [PATCH 40/57] libmultipath: fixup dm_rename to complete cookie on failure

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:41PM +0200, Hannes Reinecke wrote:

I usually try to follow the lead of the cookie using-code of the lvm2
library, and that doesn't call udev_complete() on dm_task_set_cookie()
failures. However, it does look like there is a possible problem with

dm_task_set_cookie -> _udev_notify_sem_inc -> semctl(semid, 0, GETVAL)

or

dm_task_set_cookie -> _udev_notify_sem_create -> semctl(gen_semid, 0,
GETVAL) or later

failing.  If that happens, the semaphore stays incremented, and
dm_task_set_cookie returns failure.  In the general case it could be
risky to call udev_complete when dm_task_set_cookie failed. The 
semaphore might have not gotten incremented, and if we were doing
multiple operations with the cookie, we might not wait for all of them
any more. But the way multipath is using cookies, I think this is
always safe. At worst, it won't be necessary, and we'll do nothing.

But this should probably get fixed in the libdevmapper code, so that
when dm_task_set_cookie fails, in cleans up any partial operations.

ACK (we can pull these back out when it's been sorted in libdevmapper)
-Ben

> >From my understanding we should be calling udev_complete() on
> a cookie if dm_task_set_cookie() failed.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.com>
> ---
>  libmultipath/devmapper.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
> index c2ae83b..b10f9e6 100644
> --- a/libmultipath/devmapper.c
> +++ b/libmultipath/devmapper.c
> @@ -1440,8 +1440,10 @@ dm_rename (const char * old, char * new)
>  	dm_task_no_open_count(dmt);
>  
>  	if (!dm_task_set_cookie(dmt, &cookie,
> -				DM_UDEV_DISABLE_LIBRARY_FALLBACK))
> +				DM_UDEV_DISABLE_LIBRARY_FALLBACK)) {
> +		dm_udev_complete(cookie);
>  		goto out;
> +	}
>  	r = dm_task_run(dmt);
>  
>  	if (!r)
> -- 
> 2.6.6

[PATCH 41/57] multipathd: accept zero-size paths in ev_add_path()

[Hannes Reinecke]

Paths in ALUA 'standby' state are not required to support
READ CAPACITY, and hence might report a size of '0'. However,
they are valid devices and should be added to the existing
topology.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index c1b94bd..c56672a 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -560,7 +560,7 @@ ev_add_path (struct path * pp, struct vectors * vecs)
 	pp->mpp = mpp;
 rescan:
 	if (mpp) {
-		if (mpp->size != pp->size) {
+		if (pp->size && mpp->size != pp->size) {
 			condlog(0, "%s: failed to add new path %s, "
 				"device size mismatch",
 				mpp->alias, pp->dev);
-- 
2.6.6

[PATCH 42/57] Use multipath wwid if path wwid is empty

[Hannes Reinecke]

During booting udev might not be fully initialized, causing
it to not return any information for a path.
The multipath map, however, would still be present.
In these cases we should use the multipath map as the
path wwid to avoid spurious path failures during boot.
The situation will be corrected anyway as udev will
be sending out the 'real' events very soon after.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/dmparser.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index 2209b2d..c19c5d2 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -336,10 +336,16 @@ disassemble_map (vector pathvec, char * params, struct multipath * mpp)
 					goto out1;
 
 				strncpy(pp->dev_t, word, BLK_DEV_SIZE);
-
+				if (strlen(mpp->wwid)) {
+					strncpy(pp->wwid, mpp->wwid, WWID_SIZE);
+				}
 				/* Only call this in multipath client mode */
 				if (!conf->daemon && store_path(pathvec, pp))
 					goto out1;
+			} else {
+				if (!strlen(pp->wwid) &&
+				    strlen(mpp->wwid))
+					strncpy(pp->wwid, mpp->wwid, WWID_SIZE);
 			}
 			FREE(word);
 
-- 
2.6.6

[PATCH 43/57] multipathd: set uxsock_timeout after reconfiguration

[Hannes Reinecke]

We should be setting the global variable 'uxsock_timeout' after
reconfiguration, not from within the uxsock_listen() function.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipathd/main.c   |  5 +++++
 multipathd/main.h   |  1 +
 multipathd/uxlsnr.c | 15 ++++-----------
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index c56672a..ebb9a50 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -34,6 +34,8 @@
 static int use_watchdog;
 #endif
 
+int uxsock_timeout;
+
 /*
  * libmultipath
  */
@@ -1700,6 +1702,7 @@ reconfigure (struct vectors * vecs)
 	} else {
 		conf = old;
 	}
+	uxsock_timeout = conf->uxsock_timeout;
 
 	running_state = DAEMON_RUNNING;
 
@@ -1933,6 +1936,8 @@ child (void * param)
 	if (load_config(DEFAULT_CONFIGFILE, udev))
 		goto failed;
 
+	uxsock_timeout = conf->uxsock_timeout;
+
 	dm_drv_version(conf->version, TGT_MPATH);
 	if (init_checkers()) {
 		condlog(0, "failed to initialize checkers");
diff --git a/multipathd/main.h b/multipathd/main.h
index 2f706d2..d1a6d71 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -15,6 +15,7 @@ struct prout_param_descriptor;
 struct prin_resp;
 
 extern pid_t daemon_pid;
+extern int uxsock_timeout;
 
 void exit_daemon(void);
 const char * daemon_status(void);
diff --git a/multipathd/uxlsnr.c b/multipathd/uxlsnr.c
index 77efa8a..9912e00 100644
--- a/multipathd/uxlsnr.c
+++ b/multipathd/uxlsnr.c
@@ -131,7 +131,7 @@ void uxsock_cleanup(void *arg)
 void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 {
 	int ux_sock;
-	int rlen, timeout;
+	int rlen;
 	char *inbuf;
 	char *reply;
 	sigset_t mask;
@@ -164,14 +164,6 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 		struct client *c, *tmp;
 		int i, poll_count, num_clients;
 
-		/*
-		 * Store configuration timeout;
-		 * configuration might change during
-		 * the call to 'reconfigure'.
-		 */
-		if (conf)
-			timeout = conf->uxsock_timeout;
-
 		/* setup for a poll */
 		pthread_mutex_lock(&client_lock);
 		num_clients = 0;
@@ -247,7 +239,8 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 				}
 				if (gettimeofday(&start_time, NULL) != 0)
 					start_time.tv_sec = 0;
-				if (recv_packet(c->fd, &inbuf, timeout) != 0) {
+				if (recv_packet(c->fd, &inbuf,
+						uxsock_timeout) != 0) {
 					dead_client(c);
 					continue;
 				}
@@ -268,7 +261,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 					reply = NULL;
 				}
 				check_timeout(start_time, inbuf,
-					      timeout);
+					      uxsock_timeout);
 				FREE(inbuf);
 			}
 		}
-- 
2.6.6

[PATCH 44/57] multipathd: Do not switch paths on empty multipath tables

[Hannes Reinecke]

If all paths are removed from a multipath table there is no
point in trying to switch path groups as we don't have any.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipathd/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/multipathd/main.c b/multipathd/main.c
index ebb9a50..a86530f 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -126,6 +126,9 @@ need_switch_pathgroup (struct multipath * mpp, int refresh)
 			vector_foreach_slot (pgp->paths, pp, j)
 				pathinfo(pp, conf->hwtable, DI_PRIO);
 
+	if (!mpp->pg || VECTOR_SIZE(mpp->paths) == 0)
+		return 0;
+
 	mpp->bestpg = select_path_group(mpp);
 
 	if (mpp->bestpg != mpp->nextpg)
-- 
2.6.6

[PATCH 45/57] libmultipath: remove 'get_info' argument for adopt_paths()

[Hannes Reinecke]

The 'get_info' argument was always set to '1', so we can drop it.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/structs_vec.c | 10 +++++-----
 libmultipath/structs_vec.h |  2 +-
 multipathd/main.c          |  4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index d9a731a..7d46d42 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -46,7 +46,7 @@ update_mpp_paths(struct multipath * mpp, vector pathvec)
 }
 
 extern int
-adopt_paths (vector pathvec, struct multipath * mpp, int get_info)
+adopt_paths (vector pathvec, struct multipath * mpp)
 {
 	int i;
 	struct path * pp;
@@ -69,8 +69,8 @@ adopt_paths (vector pathvec, struct multipath * mpp, int get_info)
 			if (!find_path_by_dev(mpp->paths, pp->dev) &&
 			    store_path(mpp->paths, pp))
 					return 1;
-			if (get_info && pathinfo(pp, conf->hwtable,
-						 DI_PRIO | DI_CHECKER))
+			if (pathinfo(pp, conf->hwtable,
+				     DI_PRIO | DI_CHECKER))
 				return 1;
 		}
 	}
@@ -422,7 +422,7 @@ add_map_without_path (struct vectors * vecs, char * alias)
 	if (setup_multipath(vecs, mpp))
 		return NULL; /* mpp freed in setup_multipath */
 
-	if (adopt_paths(vecs->pathvec, mpp, 1))
+	if (adopt_paths(vecs->pathvec, mpp))
 		goto out;
 
 	if (!vector_alloc_slot(vecs->mpvec))
@@ -474,7 +474,7 @@ add_map_with_path (struct vectors * vecs,
 		goto out;
 	mpp->size = pp->size;
 
-	if (adopt_paths(vecs->pathvec, mpp, 1))
+	if (adopt_paths(vecs->pathvec, mpp))
 		goto out;
 
 	if (add_vec) {
diff --git a/libmultipath/structs_vec.h b/libmultipath/structs_vec.h
index eb8e672..2e06beb 100644
--- a/libmultipath/structs_vec.h
+++ b/libmultipath/structs_vec.h
@@ -15,7 +15,7 @@ struct vectors {
 
 void set_no_path_retry(struct multipath *mpp);
 
-int adopt_paths (vector pathvec, struct multipath * mpp, int get_info);
+int adopt_paths (vector pathvec, struct multipath * mpp);
 void orphan_paths (vector pathvec, struct multipath * mpp);
 void orphan_path (struct path * pp, const char *reason);
 
diff --git a/multipathd/main.c b/multipathd/main.c
index a86530f..1141fb6 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -275,7 +275,7 @@ update_map (struct multipath *mpp, struct vectors *vecs)
 
 retry:
 	condlog(4, "%s: updating new map", mpp->alias);
-	if (adopt_paths(vecs->pathvec, mpp, 1)) {
+	if (adopt_paths(vecs->pathvec, mpp)) {
 		condlog(0, "%s: failed to adopt paths for new map update",
 			mpp->alias);
 		retries = -1;
@@ -578,7 +578,7 @@ rescan:
 
 		condlog(4,"%s: adopting all paths for path %s",
 			mpp->alias, pp->dev);
-		if (adopt_paths(vecs->pathvec, mpp, 1))
+		if (adopt_paths(vecs->pathvec, mpp))
 			goto fail; /* leave path added to pathvec */
 
 		verify_paths(mpp, vecs);
-- 
2.6.6

[PATCH 46/57] libmultipath: ensure 'dev_t' is set when store paths

[Hannes Reinecke]

We need both, 'pp->dev' and 'pp->dev_t' for correct lookup to
work. So ensure dev_t is always being updated when storing a new path.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/discovery.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 2cf2566..db82b00 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -1599,6 +1599,16 @@ pathinfo (struct path *pp, vector hwtable, int mask)
 	condlog(3, "%s: mask = 0x%x", pp->dev, mask);
 
 	/*
+	 * Sanity check: we need the device number to
+	 * avoid inconsistent information in
+	 * find_path_by_dev()/find_path_by_devt()
+	 */
+	if (!strlen(pp->dev_t) && !(mask & DI_SYSFS)) {
+		condlog(1, "%s: empty device number", pp->dev);
+		mask |= DI_SYSFS;
+	}
+
+	/*
 	 * fetch info available in sysfs
 	 */
 	if (mask & DI_SYSFS && sysfs_pathinfo(pp))
-- 
2.6.6

[PATCH 47/57] libmultipath: sanity check on store_path()

[Hannes Reinecke]

If a path has neither pp->dev nor pp->dev_t set we cannot
identify the path at all.
So reject any path where this is not the case.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/structs.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/libmultipath/structs.c b/libmultipath/structs.c
index 39b2e05..937e4a2 100644
--- a/libmultipath/structs.c
+++ b/libmultipath/structs.c
@@ -283,6 +283,20 @@ free_multipathvec (vector mpvec, enum free_path_mode free_paths)
 int
 store_path (vector pathvec, struct path * pp)
 {
+	int err = 0;
+
+	if (!strlen(pp->dev_t)) {
+		condlog(2, "%s: Empty device number", pp->dev);
+		err++;
+	}
+	if (!strlen(pp->dev)) {
+		condlog(2, "%s: Empty device name", pp->dev_t);
+		err++;
+	}
+
+	if (err > 1)
+		return 1;
+
 	if (!vector_alloc_slot(pathvec))
 		return 1;
 
-- 
2.6.6

[PATCH 48/57] dmparser: Use find_path_by_dev()

[Hannes Reinecke]

There are two functions for looking up paths, find_path_by_dev() and
find_path_by_devt(). To avoid any inconsistencies we should be
using find_path_by_dev() here and allocate a path with both values.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 libmultipath/configure.c |  6 +++++-
 libmultipath/dmparser.c  | 12 +++++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/libmultipath/configure.c b/libmultipath/configure.c
index dee749b..ed6cf98 100644
--- a/libmultipath/configure.c
+++ b/libmultipath/configure.c
@@ -969,7 +969,11 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)
 
 	if (dev_type == DEV_DEVT) {
 		strchop(dev);
-		pp = find_path_by_devt(pathvec, dev);
+		if (devt2devname(buff, FILE_NAME_SIZE, dev)) {
+			condlog(0, "%s: cannot find block device\n", dev);
+			return 1;
+		}
+		pp = find_path_by_dev(pathvec, buff);
 		if (!pp) {
 			struct udev_device *udevice = udev_device_new_from_devnum(conf->udev, 'b', parse_devt(dev));
 
diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index c19c5d2..289be89 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -320,14 +320,23 @@ disassemble_map (vector pathvec, char * params, struct multipath * mpp)
 		FREE(word);
 
 		for (j = 0; j < num_paths; j++) {
+			char devname[FILE_NAME_SIZE];
+
 			pp = NULL;
 			p += get_word(p, &word);
 
 			if (!word)
 				goto out;
 
+			if (devt2devname(devname, FILE_NAME_SIZE, word)) {
+				condlog(2, "%s: cannot find block device",
+					word);
+				FREE(word);
+				continue;
+			}
+
 			if (pathvec)
-				pp = find_path_by_devt(pathvec, word);
+				pp = find_path_by_dev(pathvec, devname);
 
 			if (!pp) {
 				pp = alloc_path();
@@ -336,6 +345,7 @@ disassemble_map (vector pathvec, char * params, struct multipath * mpp)
 					goto out1;
 
 				strncpy(pp->dev_t, word, BLK_DEV_SIZE);
+				strncpy(pp->dev, devname, FILE_NAME_SIZE);
 				if (strlen(mpp->wwid)) {
 					strncpy(pp->wwid, mpp->wwid, WWID_SIZE);
 				}
-- 
2.6.6

[PATCH 49/57] multipathd: do not flush maps on startup

[Hannes Reinecke]

When the daemon is started prior to udev the paths are not present
(yet). However, the maps themselves will be read from device-mapper.
This causes existing maps to be dropped in coalesce_maps(), only
to be reinstated later when all paths are discovered.
To avoid this we should be checking for a valid hardware entry
per multipath map; if there is none we're in startup and shouldn't
drop any maps.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipathd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 1141fb6..33a3503 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -155,7 +155,7 @@ coalesce_maps(struct vectors *vecs, vector nmpv)
 
 	vector_foreach_slot (ompv, ompp, i) {
 		condlog(3, "%s: coalesce map", ompp->alias);
-		if (!find_mp_by_wwid(nmpv, ompp->wwid)) {
+		if (ompp->hwe && !find_mp_by_wwid(nmpv, ompp->wwid)) {
 			/*
 			 * remove all current maps not allowed by the
 			 * current configuration
-- 
2.6.6

Re: [PATCH 49/57] multipathd: do not flush maps on startup

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:50PM +0200, Hannes Reinecke wrote:
> When the daemon is started prior to udev the paths are not present
> (yet). However, the maps themselves will be read from device-mapper.
> This causes existing maps to be dropped in coalesce_maps(), only
> to be reinstated later when all paths are discovered.
> To avoid this we should be checking for a valid hardware entry
> per multipath map; if there is none we're in startup and shouldn't
> drop any maps.

This doesn't just skip maps when udev hasn't started yet.  It skips any
blacklisted maps.  For instance, if you create some multipath devices,
add the lines

blacklist {
	devnode ".*"
}

to /etc/multipath.conf, and restart or reload multipathd, all the
blacklisted devices are still there.

NAK

-Ben
> 
> Signed-off-by: Hannes Reinecke <hare@suse.com>
> ---
>  multipathd/main.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 1141fb6..33a3503 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -155,7 +155,7 @@ coalesce_maps(struct vectors *vecs, vector nmpv)
>  
>  	vector_foreach_slot (ompv, ompp, i) {
>  		condlog(3, "%s: coalesce map", ompp->alias);
> -		if (!find_mp_by_wwid(nmpv, ompp->wwid)) {
> +		if (ompp->hwe && !find_mp_by_wwid(nmpv, ompp->wwid)) {
>  			/*
>  			 * remove all current maps not allowed by the
>  			 * current configuration
> -- 
> 2.6.6

Re: [PATCH 49/57] multipathd: do not flush maps on startup

[Hannes Reinecke]

On 05/03/2016 04:30 AM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:50PM +0200, Hannes Reinecke wrote:
>> When the daemon is started prior to udev the paths are not present
>> (yet). However, the maps themselves will be read from device-mapper.
>> This causes existing maps to be dropped in coalesce_maps(), only
>> to be reinstated later when all paths are discovered.
>> To avoid this we should be checking for a valid hardware entry
>> per multipath map; if there is none we're in startup and shouldn't
>> drop any maps.
> 
> This doesn't just skip maps when udev hasn't started yet.  It skips any
> blacklisted maps.  For instance, if you create some multipath devices,
> add the lines
> 
> blacklist {
> 	devnode ".*"
> }
> 
> to /etc/multipath.conf, and restart or reload multipathd, all the
> blacklisted devices are still there.
> 
> NAK
> 
Okay, I'll remove this patch when I'm resending the patchset.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[PATCH 50/57] multipathd: strict loop timings

[Hannes Reinecke]

This patch implements a new configuration value 'strict_timing',
which causes each path checker loop to start at exactly one second
after the last one. Longer path checker loops will be properly
accounted, so that each path is guaranteed to be checked within
'polling_interval' seconds.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/config.h      |  1 +
 libmultipath/dict.c        |  4 +++
 multipath/multipath.conf.5 | 14 ++++++++
 multipathd/main.c          | 79 +++++++++++++++++++++++++++++++++++++++-------
 4 files changed, 87 insertions(+), 11 deletions(-)

diff --git a/libmultipath/config.h b/libmultipath/config.h
index e9828ea..466e31e 100644
--- a/libmultipath/config.h
+++ b/libmultipath/config.h
@@ -135,6 +135,7 @@ struct config {
 	int delay_watch_checks;
 	int delay_wait_checks;
 	int uxsock_timeout;
+	int strict_timing;
 	int retrigger_tries;
 	int retrigger_delay;
 	int ignore_new_devs;
diff --git a/libmultipath/dict.c b/libmultipath/dict.c
index 89d42a1..7f912ec 100644
--- a/libmultipath/dict.c
+++ b/libmultipath/dict.c
@@ -399,6 +399,9 @@ declare_def_snprint(retrigger_delay, print_int)
 declare_def_handler(uev_wait_timeout, set_int)
 declare_def_snprint(uev_wait_timeout, print_int)
 
+declare_def_handler(strict_timing, set_yes_no)
+declare_def_snprint(strict_timing, print_yes_no)
+
 static int
 def_config_dir_handler(vector strvec)
 {
@@ -1365,6 +1368,7 @@ init_keywords(void)
 	install_keyword("retain_attached_hw_handler", &def_retain_hwhandler_handler, &snprint_def_retain_hwhandler);
 	install_keyword("detect_prio", &def_detect_prio_handler, &snprint_def_detect_prio);
 	install_keyword("force_sync", &def_force_sync_handler, &snprint_def_force_sync);
+	install_keyword("strict_timing", &def_strict_timing_handler, &snprint_def_strict_timing);
 	install_keyword("deferred_remove", &def_deferred_remove_handler, &snprint_def_deferred_remove);
 	install_keyword("partition_delimiter", &def_partition_delim_handler, &snprint_def_partition_delim);
 	install_keyword("config_dir", &def_config_dir_handler, &snprint_def_config_dir);
diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5
index b21279e..2ff88c4 100644
--- a/multipath/multipath.conf.5
+++ b/multipath/multipath.conf.5
@@ -498,6 +498,20 @@ used until it has passed
 checks. Default is
 .I no
 .TP
+.B strict_timinig
+If set to
+.I yes
+, multipathd will start a new path checker loop after exactly one second,
+so that each path check will occur at exactly
+.I polling_interval
+seconds. On busy systems path checks might take longer than one second;
+here the missing ticks will be accounted for on the next round.
+A warning will be printed if path checks take longer than
+.I polling_interval
+seconds.
+Default is
+.I no
+.TP
 .B uxsock_timeout
 CLI receive timeout in milliseconds. For larger systems CLI commands
 might timeout before the multipathd lock is released and the CLI command
diff --git a/multipathd/main.c b/multipathd/main.c
index 33a3503..d8fa88f 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1260,7 +1260,7 @@ int update_path_groups(struct multipath *mpp, struct vectors *vecs, int refresh)
  * Returns '1' if the path has been checked, '0' otherwise
  */
 int
-check_path (struct vectors * vecs, struct path * pp)
+check_path (struct vectors * vecs, struct path * pp, int ticks)
 {
 	int newstate;
 	int new_path_up = 0;
@@ -1273,7 +1273,9 @@ check_path (struct vectors * vecs, struct path * pp)
 	     pp->initialized == INIT_REQUESTED_UDEV) && !pp->mpp)
 		return 0;
 
-	if (pp->tick && --pp->tick)
+	if (pp->tick)
+		pp->tick -= (pp->tick > ticks) ? ticks : pp->tick;
+	if (pp->tick)
 		return 0; /* don't check this path yet */
 
 	if (!pp->mpp && pp->initialized == INIT_MISSING_UDEV &&
@@ -1515,6 +1517,8 @@ checkerloop (void *ap)
 	struct path *pp;
 	int count = 0;
 	unsigned int i;
+	struct itimerval timer_tick_it;
+	struct timeval last_time;
 
 	mlockall(MCL_CURRENT | MCL_FUTURE);
 	vecs = (struct vectors *)ap;
@@ -1527,23 +1531,41 @@ checkerloop (void *ap)
 		pp->checkint = conf->checkint;
 	}
 
+	/* Tweak start time for initial path check */
+	if (gettimeofday(&last_time, NULL) != 0)
+		last_time.tv_sec = 0;
+	else
+		last_time.tv_sec -= 1;
+
 	while (1) {
 		struct timeval diff_time, start_time, end_time;
-		int num_paths = 0;
+		int num_paths = 0, ticks = 0, signo, strict_timing;
+		sigset_t mask;
 
 		if (gettimeofday(&start_time, NULL) != 0)
 			start_time.tv_sec = 0;
 		pthread_cleanup_push(cleanup_lock, &vecs->lock);
 		lock(vecs->lock);
 		pthread_testcancel();
-		condlog(4, "tick");
+		strict_timing = conf->strict_timing;
+		if (start_time.tv_sec && last_time.tv_sec) {
+			timersub(&start_time, &last_time, &diff_time);
+			condlog(4, "tick (%lu.%06lu secs)",
+				diff_time.tv_sec, diff_time.tv_usec);
+			last_time.tv_sec = start_time.tv_sec;
+			last_time.tv_usec = start_time.tv_usec;
+			ticks = diff_time.tv_sec;
+		} else {
+			ticks = 1;
+			condlog(4, "tick (%d ticks)", ticks);
+		}
 #ifdef USE_SYSTEMD
 		if (use_watchdog)
 			sd_notify(0, "WATCHDOG=1");
 #endif
 		if (vecs->pathvec) {
 			vector_foreach_slot (vecs->pathvec, pp, i) {
-				num_paths += check_path(vecs, pp);
+				num_paths += check_path(vecs, pp, ticks);
 			}
 		}
 		if (vecs->mpvec) {
@@ -1560,15 +1582,49 @@ checkerloop (void *ap)
 		}
 
 		lock_cleanup_pop(vecs->lock);
+		diff_time.tv_usec = 0;
 		if (start_time.tv_sec &&
-		    gettimeofday(&end_time, NULL) == 0 &&
-		    num_paths) {
+		    gettimeofday(&end_time, NULL)) {
 			timersub(&end_time, &start_time, &diff_time);
-			condlog(3, "checked %d path%s in %lu.%06lu secs",
-				num_paths, num_paths > 1 ? "s" : "",
-				diff_time.tv_sec, diff_time.tv_usec);
+			if (num_paths) {
+				condlog(3, "checked %d path%s in %lu.%06lu secs",
+					num_paths, num_paths > 1 ? "s" : "",
+					diff_time.tv_sec, diff_time.tv_usec);
+				if (diff_time.tv_sec > conf->max_checkint)
+					condlog(1, "path checkers took longer "
+						"than %lu seconds, consider "
+						"increasing max_polling_interval",
+						diff_time.tv_sec);
+			}
+		}
+
+		if (!strict_timing)
+			sleep(1);
+		else {
+			timer_tick_it.it_interval.tv_sec = 0;
+			timer_tick_it.it_interval.tv_usec = 0;
+			if (diff_time.tv_usec) {
+				timer_tick_it.it_value.tv_sec = 0;
+				timer_tick_it.it_value.tv_usec =
+					(unsigned long)1000000 - diff_time.tv_usec;
+			} else {
+				timer_tick_it.it_value.tv_sec = 1;
+				timer_tick_it.it_value.tv_usec = 0;
+			}
+			setitimer(ITIMER_REAL, &timer_tick_it, NULL);
+
+			sigemptyset(&mask);
+			sigaddset(&mask, SIGALRM);
+			condlog(3, "waiting for %lu.%06lu secs",
+				timer_tick_it.it_value.tv_sec,
+				timer_tick_it.it_value.tv_usec);
+			if (sigwait(&mask, &signo) != 0) {
+				condlog(3, "sigwait failed with error %d",
+					errno);
+				conf->strict_timing = 0;
+				break;
+			}
 		}
-		sleep(1);
 	}
 	return NULL;
 }
@@ -1819,6 +1875,7 @@ signal_init(void)
 	sigaddset(&set, SIGHUP);
 	sigaddset(&set, SIGUSR1);
 	sigaddset(&set, SIGUSR2);
+	sigaddset(&set, SIGALRM);
 	pthread_sigmask(SIG_BLOCK, &set, NULL);
 
 	signal_set(SIGHUP, sighup);
-- 
2.6.6

[PATCH 51/57] multipathd: Provide standard error description on cli failure

[Hannes Reinecke]

When the cli command fails we should be returning the error
in addition to the standard help text.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/cli.c | 53 +++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 37 insertions(+), 16 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 6a5c6db..ab1365b 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Christophe Varoqui
  */
+#include <errno.h>
 #include <memory.h>
 #include <vector.h>
 #include <parser.h>
@@ -224,10 +225,14 @@ find_key (const char * str)
 	return foundkw;
 }
 
-#define E_SYNTAX	1
-#define E_NOPARM	2
-#define E_NOMEM		3
-
+/*
+ * get_cmdvec
+ *
+ * returns:
+ * ENOMEM: not enough memory to allocate command
+ * EAGAIN: command not found
+ * EINVAL: argument missing for command
+ */
 static int
 get_cmdvec (char * cmd, vector *v)
 {
@@ -241,13 +246,13 @@ get_cmdvec (char * cmd, vector *v)
 
 	strvec = alloc_strvec(cmd);
 	if (!strvec)
-		return E_NOMEM;
+		return ENOMEM;
 
 	cmdvec = vector_alloc();
 
 	if (!cmdvec) {
 		free_strvec(strvec);
-		return E_NOMEM;
+		return ENOMEM;
 	}
 
 	vector_foreach_slot(strvec, buff, i) {
@@ -260,17 +265,17 @@ get_cmdvec (char * cmd, vector *v)
 		}
 		kw = find_key(buff);
 		if (!kw) {
-			r = E_SYNTAX;
+			r = EAGAIN;
 			goto out;
 		}
 		cmdkw = alloc_key();
 		if (!cmdkw) {
-			r = E_NOMEM;
+			r = ENOMEM;
 			goto out;
 		}
 		if (!vector_alloc_slot(cmdvec)) {
 			FREE(cmdkw);
-			r = E_NOMEM;
+			r = ENOMEM;
 			goto out;
 		}
 		vector_set_slot(cmdvec, cmdkw);
@@ -280,7 +285,7 @@ get_cmdvec (char * cmd, vector *v)
 			get_param = 1;
 	}
 	if (get_param) {
-		r = E_NOPARM;
+		r = EINVAL;
 		goto out;
 	}
 	*v = cmdvec;
@@ -340,13 +345,29 @@ genhelp_sprint_aliases (char * reply, int maxlen, vector keys,
 }
 
 static int
-do_genhelp(char *reply, int maxlen) {
+do_genhelp(char *reply, int maxlen, const char *cmd, int error) {
 	int len = 0;
 	int i, j;
 	uint64_t fp;
 	struct handler * h;
 	struct key * kw;
 
+	switch(error) {
+	case ENOMEM:
+		len += snprintf(reply + len, maxlen - len,
+				"%s: Not enough memory\n", cmd);
+		break;
+	case EAGAIN:
+		len += snprintf(reply + len, maxlen - len,
+				"%s: not found\n", cmd);
+		break;
+	case EINVAL:
+		len += snprintf(reply + len, maxlen - len,
+				"%s: Missing argument\n", cmd);
+		break;
+	}
+	if (len >= maxlen)
+		goto out;
 	len += snprintf(reply + len, maxlen - len, VERSION_STRING);
 	if (len >= maxlen)
 		goto out;
@@ -388,7 +409,7 @@ out:
 
 
 static char *
-genhelp_handler (void)
+genhelp_handler (const char *cmd, int error)
 {
 	char * reply;
 	char * p = NULL;
@@ -401,7 +422,7 @@ genhelp_handler (void)
 		if (!reply)
 			return NULL;
 		p = reply;
-		p += do_genhelp(reply, maxlen);
+		p += do_genhelp(reply, maxlen, cmd, error);
 		again = ((p - reply) >= maxlen);
 		REALLOC_REPLY(reply, again, maxlen);
 	}
@@ -418,7 +439,7 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)
 	r = get_cmdvec(cmd, &cmdvec);
 
 	if (r) {
-		*reply = genhelp_handler();
+		*reply = genhelp_handler(cmd, r);
 		*len = strlen(*reply) + 1;
 		return 0;
 	}
@@ -426,7 +447,7 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)
 	h = find_handler(fingerprint(cmdvec));
 
 	if (!h || !h->fn) {
-		*reply = genhelp_handler();
+		*reply = genhelp_handler(cmd, EINVAL);
 		*len = strlen(*reply) + 1;
 		free_keys(cmdvec);
 		return 0;
@@ -562,7 +583,7 @@ key_generator (const char * str, int state)
 		/*
 		 * If last keyword takes a param, don't even try to guess
 		 */
-		if (r == E_NOPARM) {
+		if (r == EINVAL) {
 			has_param = 1;
 			return (strdup("(value)"));
 		}
-- 
2.6.6

[PATCH 52/57] libmultipath: make 'dm_addmap' static

[Hannes Reinecke]

dm_addmap() is only used within devmapper.c, so it can be made
static.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/devmapper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index b10f9e6..4a87bf7 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -275,7 +275,7 @@ dm_device_remove (const char *name, int needsync, int deferred_remove) {
 			    deferred_remove, needsync ? &cookie : NULL);
 }
 
-extern int
+static int
 dm_addmap (int task, const char *target, struct multipath *mpp,
 	   char * params, int ro, uint32_t *cookie) {
 	int r = 0;
-- 
2.6.6

[PATCH 53/57] multipathd: implement 'show map $map format $fmt'

[Hannes Reinecke]

Similar to the existing 'show map $map topology', but allowing
formatted content.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/cli.c          |  2 ++
 multipathd/cli_handlers.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 multipathd/cli_handlers.h |  2 ++
 multipathd/main.c         |  2 ++
 4 files changed, 73 insertions(+)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index ab1365b..3c9cdbf 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -497,6 +497,8 @@ cli_init (void) {
 	add_handler(LIST+MAPS+TOPOLOGY, NULL);
 	add_handler(LIST+TOPOLOGY, NULL);
 	add_handler(LIST+MAP+TOPOLOGY, NULL);
+	add_handler(LIST+MAP+FMT, NULL);
+	add_handler(LIST+MAP+RAW+FMT, NULL);
 	add_handler(LIST+CONFIG, NULL);
 	add_handler(LIST+BLACKLIST, NULL);
 	add_handler(LIST+DEVICES, NULL);
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index 886a5fd..dbdcbc2 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -349,6 +349,33 @@ show_daemon (char ** r, int *len)
 }
 
 int
+show_map (char ** r, int *len, struct multipath * mpp, char * style,
+	  int pretty)
+{
+	char * c;
+	char * reply;
+	unsigned int maxlen = INITIAL_REPLY_LEN;
+	int again = 1;
+
+	reply = MALLOC(maxlen);
+	while (again) {
+		if (!reply)
+			return 1;
+
+		c = reply;
+		c += snprint_multipath(c, reply + maxlen - c, style,
+				       mpp, pretty);
+
+		again = ((c - reply) == (maxlen - 1));
+
+		REALLOC_REPLY(reply, again, maxlen);
+	}
+	*r = reply;
+	*len = (int)(c - reply + 1);
+	return 0;
+}
+
+int
 show_maps (char ** r, int *len, struct vectors * vecs, char * style,
 	   int pretty)
 {
@@ -407,6 +434,46 @@ cli_list_maps_raw (void * v, char ** reply, int * len, void * data)
 }
 
 int
+cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
+{
+	struct multipath * mpp;
+	struct vectors * vecs = (struct vectors *)data;
+	char * param = get_keyparam(v, MAP);
+	char * fmt = get_keyparam(v, FMT);
+
+	param = convert_dev(param, 0);
+	get_path_layout(vecs->pathvec, 0);
+	get_multipath_layout(vecs->mpvec, 1);
+	mpp = find_mp_by_str(vecs->mpvec, param);
+	if (!mpp)
+		return 1;
+
+	condlog(3, "list map %s fmt %s (operator)", param, fmt);
+
+	return show_map(reply, len, mpp, fmt, 1);
+}
+
+int
+cli_list_map_raw (void * v, char ** reply, int * len, void * data)
+{
+	struct multipath * mpp;
+	struct vectors * vecs = (struct vectors *)data;
+	char * param = get_keyparam(v, MAP);
+	char * fmt = get_keyparam(v, FMT);
+
+	param = convert_dev(param, 0);
+	get_path_layout(vecs->pathvec, 0);
+	get_multipath_layout(vecs->mpvec, 1);
+	mpp = find_mp_by_str(vecs->mpvec, param);
+	if (!mpp)
+		return 1;
+
+	condlog(3, "list map %s fmt %s (operator)", param, fmt);
+
+	return show_map(reply, len, mpp, fmt, 0);
+}
+
+int
 cli_list_maps (void * v, char ** reply, int * len, void * data)
 {
 	struct vectors * vecs = (struct vectors *)data;
diff --git a/multipathd/cli_handlers.h b/multipathd/cli_handlers.h
index 799f8da..5d51018 100644
--- a/multipathd/cli_handlers.h
+++ b/multipathd/cli_handlers.h
@@ -7,6 +7,8 @@ int cli_list_daemon (void * v, char ** reply, int * len, void * data);
 int cli_list_maps (void * v, char ** reply, int * len, void * data);
 int cli_list_maps_fmt (void * v, char ** reply, int * len, void * data);
 int cli_list_maps_raw (void * v, char ** reply, int * len, void * data);
+int cli_list_map_fmt (void * v, char ** reply, int * len, void * data);
+int cli_list_map_raw (void * v, char ** reply, int * len, void * data);
 int cli_list_maps_status (void * v, char ** reply, int * len, void * data);
 int cli_list_maps_stats (void * v, char ** reply, int * len, void * data);
 int cli_list_map_topology (void * v, char ** reply, int * len, void * data);
diff --git a/multipathd/main.c b/multipathd/main.c
index d8fa88f..a58dae5 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -990,6 +990,8 @@ uxlsnrloop (void * ap)
 	set_handler_callback(LIST+MAPS+TOPOLOGY, cli_list_maps_topology);
 	set_handler_callback(LIST+TOPOLOGY, cli_list_maps_topology);
 	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
+	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
+	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
 	set_handler_callback(LIST+CONFIG, cli_list_config);
 	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
 	set_handler_callback(LIST+DEVICES, cli_list_devices);
-- 
2.6.6

Re: [PATCH 53/57] multipathd: implement 'show map $map format $fmt'

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:54PM +0200, Hannes Reinecke wrote:
> Similar to the existing 'show map $map topology', but allowing
> formatted content.

This touches on a patch I've thought about writing, but never got around
to.  The issue is that with show_maps_topology() we call
update_multipath() to make sure we sync our state with the kernel before
returning it, but with show_maps(), we don't. Before we could change the
format, there was no point in updating the state in show_maps (and now,
in show_map). But now that we can get all that information, it might be
worth it.

On the other hand, update_multipath() might be more than we need.
update_multipath_strings() might be enought. And unlike when we first
wrote show_maps_topology(), we call update_multipath_strings() when we
check a path now.

I'm not sure what the best answer is, but I've considered adding (or
replacing update_multipath with) update_multipath_strings in all the map
showing functions, for consistency's sake.

but besides that thought, the patch looks fine.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  multipathd/cli.c          |  2 ++
>  multipathd/cli_handlers.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++
>  multipathd/cli_handlers.h |  2 ++
>  multipathd/main.c         |  2 ++
>  4 files changed, 73 insertions(+)
> 
> diff --git a/multipathd/cli.c b/multipathd/cli.c
> index ab1365b..3c9cdbf 100644
> --- a/multipathd/cli.c
> +++ b/multipathd/cli.c
> @@ -497,6 +497,8 @@ cli_init (void) {
>  	add_handler(LIST+MAPS+TOPOLOGY, NULL);
>  	add_handler(LIST+TOPOLOGY, NULL);
>  	add_handler(LIST+MAP+TOPOLOGY, NULL);
> +	add_handler(LIST+MAP+FMT, NULL);
> +	add_handler(LIST+MAP+RAW+FMT, NULL);
>  	add_handler(LIST+CONFIG, NULL);
>  	add_handler(LIST+BLACKLIST, NULL);
>  	add_handler(LIST+DEVICES, NULL);
> diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
> index 886a5fd..dbdcbc2 100644
> --- a/multipathd/cli_handlers.c
> +++ b/multipathd/cli_handlers.c
> @@ -349,6 +349,33 @@ show_daemon (char ** r, int *len)
>  }
>  
>  int
> +show_map (char ** r, int *len, struct multipath * mpp, char * style,
> +	  int pretty)
> +{
> +	char * c;
> +	char * reply;
> +	unsigned int maxlen = INITIAL_REPLY_LEN;
> +	int again = 1;
> +
> +	reply = MALLOC(maxlen);
> +	while (again) {
> +		if (!reply)
> +			return 1;
> +
> +		c = reply;
> +		c += snprint_multipath(c, reply + maxlen - c, style,
> +				       mpp, pretty);
> +
> +		again = ((c - reply) == (maxlen - 1));
> +
> +		REALLOC_REPLY(reply, again, maxlen);
> +	}
> +	*r = reply;
> +	*len = (int)(c - reply + 1);
> +	return 0;
> +}
> +
> +int
>  show_maps (char ** r, int *len, struct vectors * vecs, char * style,
>  	   int pretty)
>  {
> @@ -407,6 +434,46 @@ cli_list_maps_raw (void * v, char ** reply, int * len, void * data)
>  }
>  
>  int
> +cli_list_map_fmt (void * v, char ** reply, int * len, void * data)
> +{
> +	struct multipath * mpp;
> +	struct vectors * vecs = (struct vectors *)data;
> +	char * param = get_keyparam(v, MAP);
> +	char * fmt = get_keyparam(v, FMT);
> +
> +	param = convert_dev(param, 0);
> +	get_path_layout(vecs->pathvec, 0);
> +	get_multipath_layout(vecs->mpvec, 1);
> +	mpp = find_mp_by_str(vecs->mpvec, param);
> +	if (!mpp)
> +		return 1;
> +
> +	condlog(3, "list map %s fmt %s (operator)", param, fmt);
> +
> +	return show_map(reply, len, mpp, fmt, 1);
> +}
> +
> +int
> +cli_list_map_raw (void * v, char ** reply, int * len, void * data)
> +{
> +	struct multipath * mpp;
> +	struct vectors * vecs = (struct vectors *)data;
> +	char * param = get_keyparam(v, MAP);
> +	char * fmt = get_keyparam(v, FMT);
> +
> +	param = convert_dev(param, 0);
> +	get_path_layout(vecs->pathvec, 0);
> +	get_multipath_layout(vecs->mpvec, 1);
> +	mpp = find_mp_by_str(vecs->mpvec, param);
> +	if (!mpp)
> +		return 1;
> +
> +	condlog(3, "list map %s fmt %s (operator)", param, fmt);
> +
> +	return show_map(reply, len, mpp, fmt, 0);
> +}
> +
> +int
>  cli_list_maps (void * v, char ** reply, int * len, void * data)
>  {
>  	struct vectors * vecs = (struct vectors *)data;
> diff --git a/multipathd/cli_handlers.h b/multipathd/cli_handlers.h
> index 799f8da..5d51018 100644
> --- a/multipathd/cli_handlers.h
> +++ b/multipathd/cli_handlers.h
> @@ -7,6 +7,8 @@ int cli_list_daemon (void * v, char ** reply, int * len, void * data);
>  int cli_list_maps (void * v, char ** reply, int * len, void * data);
>  int cli_list_maps_fmt (void * v, char ** reply, int * len, void * data);
>  int cli_list_maps_raw (void * v, char ** reply, int * len, void * data);
> +int cli_list_map_fmt (void * v, char ** reply, int * len, void * data);
> +int cli_list_map_raw (void * v, char ** reply, int * len, void * data);
>  int cli_list_maps_status (void * v, char ** reply, int * len, void * data);
>  int cli_list_maps_stats (void * v, char ** reply, int * len, void * data);
>  int cli_list_map_topology (void * v, char ** reply, int * len, void * data);
> diff --git a/multipathd/main.c b/multipathd/main.c
> index d8fa88f..a58dae5 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -990,6 +990,8 @@ uxlsnrloop (void * ap)
>  	set_handler_callback(LIST+MAPS+TOPOLOGY, cli_list_maps_topology);
>  	set_handler_callback(LIST+TOPOLOGY, cli_list_maps_topology);
>  	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
> +	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
> +	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
>  	set_handler_callback(LIST+CONFIG, cli_list_config);
>  	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
>  	set_handler_callback(LIST+DEVICES, cli_list_devices);
> -- 
> 2.6.6

[PATCH 54/57] multipathd: Increase uxclnt timeout

[Hannes Reinecke]

When calling uxclnt we need to increase the timeout, otherwise
we will not be able to get a reply if the daemon runs into an
uxlsnr timeout.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 multipathd/main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index a58dae5..77eb498 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -2293,7 +2293,7 @@ main (int argc, char *argv[])
 		case 'k':
 			if (load_config(DEFAULT_CONFIGFILE, udev_new()))
 				exit(1);
-			uxclnt(optarg, conf->uxsock_timeout);
+			uxclnt(optarg, uxsock_timeout + 100);
 			exit(0);
 		case 'B':
 			conf->bindings_read_only = 1;
@@ -2323,7 +2323,7 @@ main (int argc, char *argv[])
 			optind++;
 		}
 		c += snprintf(c, s + CMDSIZE - c, "\n");
-		uxclnt(s, conf->uxsock_timeout);
+		uxclnt(s, uxsock_timeout + 100);
 		exit(0);
 	}
 
-- 
2.6.6

[PATCH 55/57] multipathd: asynchronous configuration

[Hannes Reinecke]

For initial configuration multipathd waits until it has synchronized
with the existing setup. On larger systems this takes up quite
some time (I've measured 80 seconds on a system with 1024 paths)
causing systemd to stall and the system to fail booting.
This patch makes the initial configuration asynchronous, and
using the same codepath as the existing 'reconfigure' CLI
command.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 libmultipath/uevent.c     |  10 +--
 multipathd/cli_handlers.c |  14 ++-
 multipathd/main.c         | 219 ++++++++++++++++++++++++++++++----------------
 multipathd/main.h         |   2 +
 4 files changed, 157 insertions(+), 88 deletions(-)

diff --git a/libmultipath/uevent.c b/libmultipath/uevent.c
index 478c6ce..fbe9c44 100644
--- a/libmultipath/uevent.c
+++ b/libmultipath/uevent.c
@@ -529,8 +529,6 @@ int uevent_listen(struct udev *udev)
 	}
 
 	pthread_sigmask(SIG_SETMASK, NULL, &mask);
-	sigdelset(&mask, SIGHUP);
-	sigdelset(&mask, SIGUSR1);
 	events = 0;
 	while (1) {
 		struct uevent *uev;
@@ -561,9 +559,11 @@ int uevent_listen(struct udev *udev)
 			continue;
 		}
 		if (fdcount < 0) {
-			if (errno != EINTR)
-				condlog(0, "error receiving "
-					"uevent message: %m");
+			if (errno == EINTR)
+				continue;
+
+			condlog(0, "error receiving "
+				"uevent message: %m");
 			err = -errno;
 			break;
 		}
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index dbdcbc2..0397353 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -909,17 +909,13 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
 int
 cli_reconfigure(void * v, char ** reply, int * len, void * data)
 {
-	struct vectors * vecs = (struct vectors *)data;
-
-	if (need_to_delay_reconfig(vecs)) {
-		conf->delayed_reconfig = 1;
-		condlog(2, "delaying reconfigure (operator)");
-		return 0;
-	}
-
 	condlog(2, "reconfigure (operator)");
 
-	return reconfigure(vecs);
+	if (set_config_state(DAEMON_CONFIGURE) == ETIMEDOUT) {
+		condlog(2, "timeout starting reconfiguration");
+		return 1;
+	}
+	return 0;
 }
 
 int
diff --git a/multipathd/main.c b/multipathd/main.c
index 77eb498..41b5a49 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -97,10 +97,11 @@ struct mpath_event_param
 unsigned int mpath_mx_alloc_len;
 
 int logsink;
-enum daemon_status running_state;
+enum daemon_status running_state = DAEMON_INIT;
 pid_t daemon_pid;
+pthread_mutex_t config_lock = PTHREAD_MUTEX_INITIALIZER;
+pthread_cond_t config_cond = PTHREAD_COND_INITIALIZER;
 
-static sem_t exit_sem;
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -108,6 +109,94 @@ struct vectors * gvecs;
 
 struct udev * udev;
 
+const char *
+daemon_status(void)
+{
+	switch (running_state) {
+	case DAEMON_INIT:
+		return "init";
+	case DAEMON_START:
+		return "startup";
+	case DAEMON_CONFIGURE:
+		return "configure";
+	case DAEMON_IDLE:
+		return "idle";
+	case DAEMON_RUNNING:
+		return "running";
+	case DAEMON_SHUTDOWN:
+		return "shutdown";
+	}
+	return NULL;
+}
+
+/*
+ * I love you too, systemd ...
+ */
+const char *
+sd_notify_status(void)
+{
+	switch (running_state) {
+	case DAEMON_INIT:
+		return "STATUS=init";
+	case DAEMON_START:
+		return "STATUS=startup";
+	case DAEMON_CONFIGURE:
+		return "STATUS=configure";
+	case DAEMON_IDLE:
+		return "STATUS=idle";
+	case DAEMON_RUNNING:
+		return "STATUS=running";
+	case DAEMON_SHUTDOWN:
+		return "STATUS=shutdown";
+	}
+	return NULL;
+}
+
+static void config_cleanup(void *arg)
+{
+	pthread_mutex_unlock(&config_lock);
+}
+
+void post_config_state(enum daemon_status state)
+{
+	pthread_mutex_lock(&config_lock);
+	if (state != running_state) {
+		running_state = state;
+		pthread_cond_broadcast(&config_cond);
+#ifdef USE_SYSTEMD
+		sd_notify(0, sd_notify_status());
+#endif
+	}
+	pthread_mutex_unlock(&config_lock);
+}
+
+int set_config_state(enum daemon_status state)
+{
+	int rc = 0;
+
+	pthread_cleanup_push(config_cleanup, NULL);
+	pthread_mutex_lock(&config_lock);
+	if (running_state != state) {
+		if (running_state != DAEMON_IDLE) {
+			struct timespec ts;
+
+			clock_gettime(CLOCK_REALTIME, &ts);
+			ts.tv_sec += 1;
+			rc = pthread_cond_timedwait(&config_cond,
+						    &config_lock, &ts);
+		}
+		if (!rc) {
+			running_state = state;
+			pthread_cond_broadcast(&config_cond);
+#ifdef USE_SYSTEMD
+			sd_notify(0, sd_notify_status());
+#endif
+		}
+	}
+	pthread_cleanup_pop(1);
+	return rc;
+}
+
 static int
 need_switch_pathgroup (struct multipath * mpp, int refresh)
 {
@@ -352,7 +441,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
 	if (mpp) {
 		if (mpp->wait_for_udev > 1) {
 			if (update_map(mpp, vecs))
-			/* setup multipathd removed the map */
+				/* setup multipathd removed the map */
 				return 1;
 		}
 		if (mpp->wait_for_udev) {
@@ -360,7 +449,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
 			if (conf->delayed_reconfig &&
 			    !need_to_delay_reconfig(vecs)) {
 				condlog(2, "reconfigure (delayed)");
-				reconfigure(vecs);
+				set_config_state(DAEMON_CONFIGURE);
 				return 0;
 			}
 		}
@@ -903,6 +992,16 @@ uev_trigger (struct uevent * uev, void * trigger_data)
 	if (uev_discard(uev->devpath))
 		return 0;
 
+	pthread_cleanup_push(config_cleanup, NULL);
+	pthread_mutex_lock(&config_lock);
+	if (running_state != DAEMON_IDLE &&
+	    running_state != DAEMON_RUNNING)
+		pthread_cond_wait(&config_cond, &config_lock);
+	pthread_cleanup_pop(1);
+
+	if (running_state == DAEMON_SHUTDOWN)
+		return 0;
+
 	pthread_cleanup_push(cleanup_lock, &vecs->lock);
 	lock(vecs->lock);
 	pthread_testcancel();
@@ -1031,25 +1130,7 @@ uxlsnrloop (void * ap)
 void
 exit_daemon (void)
 {
-	sem_post(&exit_sem);
-}
-
-const char *
-daemon_status(void)
-{
-	switch (running_state) {
-	case DAEMON_INIT:
-		return "init";
-	case DAEMON_START:
-		return "startup";
-	case DAEMON_CONFIGURE:
-		return "configure";
-	case DAEMON_RUNNING:
-		return "running";
-	case DAEMON_SHUTDOWN:
-		return "shutdown";
-	}
-	return NULL;
+	post_config_state(DAEMON_SHUTDOWN);
 }
 
 static void
@@ -1178,7 +1259,7 @@ missing_uev_wait_tick(struct vectors *vecs)
 	if (timed_out && conf->delayed_reconfig &&
 	    !need_to_delay_reconfig(vecs)) {
 		condlog(2, "reconfigure (delayed)");
-		reconfigure(vecs);
+		set_config_state(DAEMON_CONFIGURE);
 	}
 }
 
@@ -1541,11 +1622,18 @@ checkerloop (void *ap)
 
 	while (1) {
 		struct timeval diff_time, start_time, end_time;
-		int num_paths = 0, ticks = 0, signo, strict_timing;
+		int num_paths = 0, ticks = 0, signo, strict_timing, rc = 0;
 		sigset_t mask;
 
 		if (gettimeofday(&start_time, NULL) != 0)
 			start_time.tv_sec = 0;
+
+		rc = set_config_state(DAEMON_RUNNING);
+		if (rc == ETIMEDOUT) {
+			condlog(4, "timeout waiting for DAEMON_IDLE");
+			continue;
+		}
+
 		pthread_cleanup_push(cleanup_lock, &vecs->lock);
 		lock(vecs->lock);
 		pthread_testcancel();
@@ -1600,6 +1688,7 @@ checkerloop (void *ap)
 			}
 		}
 
+		post_config_state(DAEMON_IDLE);
 		if (!strict_timing)
 			sleep(1);
 		else {
@@ -1734,8 +1823,6 @@ reconfigure (struct vectors * vecs)
 	struct config * old = conf;
 	int retval = 1;
 
-	running_state = DAEMON_CONFIGURE;
-
 	/*
 	 * free old map and path vectors ... they use old conf state
 	 */
@@ -1765,8 +1852,6 @@ reconfigure (struct vectors * vecs)
 	}
 	uxsock_timeout = conf->uxsock_timeout;
 
-	running_state = DAEMON_RUNNING;
-
 	return retval;
 }
 
@@ -1819,20 +1904,9 @@ signal_set(int signo, void (*func) (int))
 void
 handle_signals(void)
 {
-	if (reconfig_sig && running_state == DAEMON_RUNNING) {
-		pthread_cleanup_push(cleanup_lock,
-				&gvecs->lock);
-		lock(gvecs->lock);
-		pthread_testcancel();
-		if (need_to_delay_reconfig(gvecs)) {
-			conf->delayed_reconfig = 1;
-			condlog(2, "delaying reconfigure (signal)");
-		}
-		else {
-			condlog(2, "reconfigure (signal)");
-			reconfigure(gvecs);
-		}
-		lock_cleanup_pop(gvecs->lock);
+	if (reconfig_sig) {
+		condlog(2, "reconfigure (signal)");
+		set_config_state(DAEMON_CONFIGURE);
 	}
 	if (log_reset_sig) {
 		condlog(2, "reset log (signal)");
@@ -1966,7 +2040,6 @@ child (void * param)
 	char *envp;
 
 	mlockall(MCL_CURRENT | MCL_FUTURE);
-	sem_init(&exit_sem, 0, 0);
 	signal_init();
 
 	udev = udev_new();
@@ -1987,11 +2060,8 @@ child (void * param)
 		exit(1);
 	}
 
-	running_state = DAEMON_START;
+	post_config_state(DAEMON_START);
 
-#ifdef USE_SYSTEMD
-	sd_notify(0, "STATUS=startup");
-#endif
 	condlog(2, "--------start up--------");
 	condlog(2, "read " DEFAULT_CONFIGFILE);
 
@@ -2067,6 +2137,11 @@ child (void * param)
 	}
 #endif
 	/*
+	 * Signal start of configuration
+	 */
+	post_config_state(DAEMON_CONFIGURE);
+
+	/*
 	 * Start uevent listener early to catch events
 	 */
 	if ((rc = pthread_create(&uevent_thr, &uevent_attr, ueventloop, udev))) {
@@ -2078,21 +2153,6 @@ child (void * param)
 		condlog(0, "failed to create cli listener: %d", rc);
 		goto failed;
 	}
-	/*
-	 * fetch and configure both paths and multipaths
-	 */
-#ifdef USE_SYSTEMD
-	sd_notify(0, "STATUS=configure");
-#endif
-	running_state = DAEMON_CONFIGURE;
-
-	lock(vecs->lock);
-	if (configure(vecs, 1)) {
-		unlock(vecs->lock);
-		condlog(0, "failure during configuration");
-		goto failed;
-	}
-	unlock(vecs->lock);
 
 	/*
 	 * start threads
@@ -2107,20 +2167,32 @@ child (void * param)
 	}
 	pthread_attr_destroy(&misc_attr);
 
-	running_state = DAEMON_RUNNING;
 #ifdef USE_SYSTEMD
-	sd_notify(0, "READY=1\nSTATUS=running");
+	sd_notify(0, "READY=1");
 #endif
 
-	/*
-	 * exit path
-	 */
-	while(sem_wait(&exit_sem) != 0); /* Do nothing */
+	while (running_state != DAEMON_SHUTDOWN) {
+		pthread_cleanup_push(config_cleanup, NULL);
+		pthread_mutex_lock(&config_lock);
+		if (running_state != DAEMON_CONFIGURE &&
+		    running_state != DAEMON_SHUTDOWN) {
+			pthread_cond_wait(&config_cond, &config_lock);
+		}
+		pthread_cleanup_pop(1);
+		if (running_state == DAEMON_CONFIGURE) {
+			pthread_cleanup_push(cleanup_lock, &vecs->lock);
+			lock(vecs->lock);
+			pthread_testcancel();
+			if (!need_to_delay_reconfig(vecs)) {
+				reconfigure(vecs);
+			} else {
+				conf->delayed_reconfig = 1;
+			}
+			lock_cleanup_pop(vecs->lock);
+			post_config_state(DAEMON_IDLE);
+		}
+	}
 
-#ifdef USE_SYSTEMD
-	sd_notify(0, "STATUS=shutdown");
-#endif
-	running_state = DAEMON_SHUTDOWN;
 	lock(vecs->lock);
 	if (conf->queue_without_daemon == QUE_NO_DAEMON_OFF)
 		vector_foreach_slot(vecs->mpvec, mpp, i)
@@ -2253,7 +2325,6 @@ main (int argc, char *argv[])
 	int foreground = 0;
 
 	logsink = 1;
-	running_state = DAEMON_INIT;
 	dm_init();
 
 	if (getuid() != 0) {
diff --git a/multipathd/main.h b/multipathd/main.h
index d1a6d71..10b3a6d 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -7,6 +7,7 @@ enum daemon_status {
     DAEMON_INIT,
     DAEMON_START,
     DAEMON_CONFIGURE,
+    DAEMON_IDLE,
     DAEMON_RUNNING,
     DAEMON_SHUTDOWN,
 };
@@ -26,6 +27,7 @@ int ev_remove_path (struct path *, struct vectors *);
 int ev_add_map (char *, char *, struct vectors *);
 int ev_remove_map (char *, char *, int, struct vectors *);
 void sync_map_state (struct multipath *);
+int set_config_state(enum daemon_status);
 void * mpath_alloc_prin_response(int prin_sa);
 int prin_do_scsi_ioctl(char *, int rq_servact, struct prin_resp * resp,
        int noisy);
-- 
2.6.6

Re: [PATCH 55/57] multipathd: asynchronous configuration

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:56PM +0200, Hannes Reinecke wrote:
> For initial configuration multipathd waits until it has synchronized
> with the existing setup. On larger systems this takes up quite
> some time (I've measured 80 seconds on a system with 1024 paths)
> causing systemd to stall and the system to fail booting.
> This patch makes the initial configuration asynchronous, and
> using the same codepath as the existing 'reconfigure' CLI
> command.

After you call reconfigure, you need to clear conf->delayed_reconfig,
otherwise ev_add_map will keep triggering reconfigures whenever a new
map is added.  This isn't actually your bug.  I forgot to do that when I
added the delayed_reconfig code in the first place. And since the patch
is already accepted, there's no reason to mess with it.

I'll send a fix.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/uevent.c     |  10 +--
>  multipathd/cli_handlers.c |  14 ++-
>  multipathd/main.c         | 219 ++++++++++++++++++++++++++++++----------------
>  multipathd/main.h         |   2 +
>  4 files changed, 157 insertions(+), 88 deletions(-)
> 
> diff --git a/libmultipath/uevent.c b/libmultipath/uevent.c
> index 478c6ce..fbe9c44 100644
> --- a/libmultipath/uevent.c
> +++ b/libmultipath/uevent.c
> @@ -529,8 +529,6 @@ int uevent_listen(struct udev *udev)
>  	}
>  
>  	pthread_sigmask(SIG_SETMASK, NULL, &mask);
> -	sigdelset(&mask, SIGHUP);
> -	sigdelset(&mask, SIGUSR1);
>  	events = 0;
>  	while (1) {
>  		struct uevent *uev;
> @@ -561,9 +559,11 @@ int uevent_listen(struct udev *udev)
>  			continue;
>  		}
>  		if (fdcount < 0) {
> -			if (errno != EINTR)
> -				condlog(0, "error receiving "
> -					"uevent message: %m");
> +			if (errno == EINTR)
> +				continue;
> +
> +			condlog(0, "error receiving "
> +				"uevent message: %m");
>  			err = -errno;
>  			break;
>  		}
> diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
> index dbdcbc2..0397353 100644
> --- a/multipathd/cli_handlers.c
> +++ b/multipathd/cli_handlers.c
> @@ -909,17 +909,13 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
>  int
>  cli_reconfigure(void * v, char ** reply, int * len, void * data)
>  {
> -	struct vectors * vecs = (struct vectors *)data;
> -
> -	if (need_to_delay_reconfig(vecs)) {
> -		conf->delayed_reconfig = 1;
> -		condlog(2, "delaying reconfigure (operator)");
> -		return 0;
> -	}
> -
>  	condlog(2, "reconfigure (operator)");
>  
> -	return reconfigure(vecs);
> +	if (set_config_state(DAEMON_CONFIGURE) == ETIMEDOUT) {
> +		condlog(2, "timeout starting reconfiguration");
> +		return 1;
> +	}
> +	return 0;
>  }
>  
>  int
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 77eb498..41b5a49 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -97,10 +97,11 @@ struct mpath_event_param
>  unsigned int mpath_mx_alloc_len;
>  
>  int logsink;
> -enum daemon_status running_state;
> +enum daemon_status running_state = DAEMON_INIT;
>  pid_t daemon_pid;
> +pthread_mutex_t config_lock = PTHREAD_MUTEX_INITIALIZER;
> +pthread_cond_t config_cond = PTHREAD_COND_INITIALIZER;
>  
> -static sem_t exit_sem;
>  /*
>   * global copy of vecs for use in sig handlers
>   */
> @@ -108,6 +109,94 @@ struct vectors * gvecs;
>  
>  struct udev * udev;
>  
> +const char *
> +daemon_status(void)
> +{
> +	switch (running_state) {
> +	case DAEMON_INIT:
> +		return "init";
> +	case DAEMON_START:
> +		return "startup";
> +	case DAEMON_CONFIGURE:
> +		return "configure";
> +	case DAEMON_IDLE:
> +		return "idle";
> +	case DAEMON_RUNNING:
> +		return "running";
> +	case DAEMON_SHUTDOWN:
> +		return "shutdown";
> +	}
> +	return NULL;
> +}
> +
> +/*
> + * I love you too, systemd ...
> + */
> +const char *
> +sd_notify_status(void)
> +{
> +	switch (running_state) {
> +	case DAEMON_INIT:
> +		return "STATUS=init";
> +	case DAEMON_START:
> +		return "STATUS=startup";
> +	case DAEMON_CONFIGURE:
> +		return "STATUS=configure";
> +	case DAEMON_IDLE:
> +		return "STATUS=idle";
> +	case DAEMON_RUNNING:
> +		return "STATUS=running";
> +	case DAEMON_SHUTDOWN:
> +		return "STATUS=shutdown";
> +	}
> +	return NULL;
> +}
> +
> +static void config_cleanup(void *arg)
> +{
> +	pthread_mutex_unlock(&config_lock);
> +}
> +
> +void post_config_state(enum daemon_status state)
> +{
> +	pthread_mutex_lock(&config_lock);
> +	if (state != running_state) {
> +		running_state = state;
> +		pthread_cond_broadcast(&config_cond);
> +#ifdef USE_SYSTEMD
> +		sd_notify(0, sd_notify_status());
> +#endif
> +	}
> +	pthread_mutex_unlock(&config_lock);
> +}
> +
> +int set_config_state(enum daemon_status state)
> +{
> +	int rc = 0;
> +
> +	pthread_cleanup_push(config_cleanup, NULL);
> +	pthread_mutex_lock(&config_lock);
> +	if (running_state != state) {
> +		if (running_state != DAEMON_IDLE) {
> +			struct timespec ts;
> +
> +			clock_gettime(CLOCK_REALTIME, &ts);
> +			ts.tv_sec += 1;
> +			rc = pthread_cond_timedwait(&config_cond,
> +						    &config_lock, &ts);
> +		}
> +		if (!rc) {
> +			running_state = state;
> +			pthread_cond_broadcast(&config_cond);
> +#ifdef USE_SYSTEMD
> +			sd_notify(0, sd_notify_status());
> +#endif
> +		}
> +	}
> +	pthread_cleanup_pop(1);
> +	return rc;
> +}
> +
>  static int
>  need_switch_pathgroup (struct multipath * mpp, int refresh)
>  {
> @@ -352,7 +441,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
>  	if (mpp) {
>  		if (mpp->wait_for_udev > 1) {
>  			if (update_map(mpp, vecs))
> -			/* setup multipathd removed the map */
> +				/* setup multipathd removed the map */
>  				return 1;
>  		}
>  		if (mpp->wait_for_udev) {
> @@ -360,7 +449,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
>  			if (conf->delayed_reconfig &&
>  			    !need_to_delay_reconfig(vecs)) {
>  				condlog(2, "reconfigure (delayed)");
> -				reconfigure(vecs);
> +				set_config_state(DAEMON_CONFIGURE);
>  				return 0;
>  			}
>  		}
> @@ -903,6 +992,16 @@ uev_trigger (struct uevent * uev, void * trigger_data)
>  	if (uev_discard(uev->devpath))
>  		return 0;
>  
> +	pthread_cleanup_push(config_cleanup, NULL);
> +	pthread_mutex_lock(&config_lock);
> +	if (running_state != DAEMON_IDLE &&
> +	    running_state != DAEMON_RUNNING)
> +		pthread_cond_wait(&config_cond, &config_lock);
> +	pthread_cleanup_pop(1);
> +
> +	if (running_state == DAEMON_SHUTDOWN)
> +		return 0;
> +
>  	pthread_cleanup_push(cleanup_lock, &vecs->lock);
>  	lock(vecs->lock);
>  	pthread_testcancel();
> @@ -1031,25 +1130,7 @@ uxlsnrloop (void * ap)
>  void
>  exit_daemon (void)
>  {
> -	sem_post(&exit_sem);
> -}
> -
> -const char *
> -daemon_status(void)
> -{
> -	switch (running_state) {
> -	case DAEMON_INIT:
> -		return "init";
> -	case DAEMON_START:
> -		return "startup";
> -	case DAEMON_CONFIGURE:
> -		return "configure";
> -	case DAEMON_RUNNING:
> -		return "running";
> -	case DAEMON_SHUTDOWN:
> -		return "shutdown";
> -	}
> -	return NULL;
> +	post_config_state(DAEMON_SHUTDOWN);
>  }
>  
>  static void
> @@ -1178,7 +1259,7 @@ missing_uev_wait_tick(struct vectors *vecs)
>  	if (timed_out && conf->delayed_reconfig &&
>  	    !need_to_delay_reconfig(vecs)) {
>  		condlog(2, "reconfigure (delayed)");
> -		reconfigure(vecs);
> +		set_config_state(DAEMON_CONFIGURE);
>  	}
>  }
>  
> @@ -1541,11 +1622,18 @@ checkerloop (void *ap)
>  
>  	while (1) {
>  		struct timeval diff_time, start_time, end_time;
> -		int num_paths = 0, ticks = 0, signo, strict_timing;
> +		int num_paths = 0, ticks = 0, signo, strict_timing, rc = 0;
>  		sigset_t mask;
>  
>  		if (gettimeofday(&start_time, NULL) != 0)
>  			start_time.tv_sec = 0;
> +
> +		rc = set_config_state(DAEMON_RUNNING);
> +		if (rc == ETIMEDOUT) {
> +			condlog(4, "timeout waiting for DAEMON_IDLE");
> +			continue;
> +		}
> +
>  		pthread_cleanup_push(cleanup_lock, &vecs->lock);
>  		lock(vecs->lock);
>  		pthread_testcancel();
> @@ -1600,6 +1688,7 @@ checkerloop (void *ap)
>  			}
>  		}
>  
> +		post_config_state(DAEMON_IDLE);
>  		if (!strict_timing)
>  			sleep(1);
>  		else {
> @@ -1734,8 +1823,6 @@ reconfigure (struct vectors * vecs)
>  	struct config * old = conf;
>  	int retval = 1;
>  
> -	running_state = DAEMON_CONFIGURE;
> -
>  	/*
>  	 * free old map and path vectors ... they use old conf state
>  	 */
> @@ -1765,8 +1852,6 @@ reconfigure (struct vectors * vecs)
>  	}
>  	uxsock_timeout = conf->uxsock_timeout;
>  
> -	running_state = DAEMON_RUNNING;
> -
>  	return retval;
>  }
>  
> @@ -1819,20 +1904,9 @@ signal_set(int signo, void (*func) (int))
>  void
>  handle_signals(void)
>  {
> -	if (reconfig_sig && running_state == DAEMON_RUNNING) {
> -		pthread_cleanup_push(cleanup_lock,
> -				&gvecs->lock);
> -		lock(gvecs->lock);
> -		pthread_testcancel();
> -		if (need_to_delay_reconfig(gvecs)) {
> -			conf->delayed_reconfig = 1;
> -			condlog(2, "delaying reconfigure (signal)");
> -		}
> -		else {
> -			condlog(2, "reconfigure (signal)");
> -			reconfigure(gvecs);
> -		}
> -		lock_cleanup_pop(gvecs->lock);
> +	if (reconfig_sig) {
> +		condlog(2, "reconfigure (signal)");
> +		set_config_state(DAEMON_CONFIGURE);
>  	}
>  	if (log_reset_sig) {
>  		condlog(2, "reset log (signal)");
> @@ -1966,7 +2040,6 @@ child (void * param)
>  	char *envp;
>  
>  	mlockall(MCL_CURRENT | MCL_FUTURE);
> -	sem_init(&exit_sem, 0, 0);
>  	signal_init();
>  
>  	udev = udev_new();
> @@ -1987,11 +2060,8 @@ child (void * param)
>  		exit(1);
>  	}
>  
> -	running_state = DAEMON_START;
> +	post_config_state(DAEMON_START);
>  
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=startup");
> -#endif
>  	condlog(2, "--------start up--------");
>  	condlog(2, "read " DEFAULT_CONFIGFILE);
>  
> @@ -2067,6 +2137,11 @@ child (void * param)
>  	}
>  #endif
>  	/*
> +	 * Signal start of configuration
> +	 */
> +	post_config_state(DAEMON_CONFIGURE);
> +
> +	/*
>  	 * Start uevent listener early to catch events
>  	 */
>  	if ((rc = pthread_create(&uevent_thr, &uevent_attr, ueventloop, udev))) {
> @@ -2078,21 +2153,6 @@ child (void * param)
>  		condlog(0, "failed to create cli listener: %d", rc);
>  		goto failed;
>  	}
> -	/*
> -	 * fetch and configure both paths and multipaths
> -	 */
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=configure");
> -#endif
> -	running_state = DAEMON_CONFIGURE;
> -
> -	lock(vecs->lock);
> -	if (configure(vecs, 1)) {
> -		unlock(vecs->lock);
> -		condlog(0, "failure during configuration");
> -		goto failed;
> -	}
> -	unlock(vecs->lock);
>  
>  	/*
>  	 * start threads
> @@ -2107,20 +2167,32 @@ child (void * param)
>  	}
>  	pthread_attr_destroy(&misc_attr);
>  
> -	running_state = DAEMON_RUNNING;
>  #ifdef USE_SYSTEMD
> -	sd_notify(0, "READY=1\nSTATUS=running");
> +	sd_notify(0, "READY=1");
>  #endif
>  
> -	/*
> -	 * exit path
> -	 */
> -	while(sem_wait(&exit_sem) != 0); /* Do nothing */
> +	while (running_state != DAEMON_SHUTDOWN) {
> +		pthread_cleanup_push(config_cleanup, NULL);
> +		pthread_mutex_lock(&config_lock);
> +		if (running_state != DAEMON_CONFIGURE &&
> +		    running_state != DAEMON_SHUTDOWN) {
> +			pthread_cond_wait(&config_cond, &config_lock);
> +		}
> +		pthread_cleanup_pop(1);
> +		if (running_state == DAEMON_CONFIGURE) {
> +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +			lock(vecs->lock);
> +			pthread_testcancel();
> +			if (!need_to_delay_reconfig(vecs)) {
> +				reconfigure(vecs);
> +			} else {
> +				conf->delayed_reconfig = 1;
> +			}
> +			lock_cleanup_pop(vecs->lock);
> +			post_config_state(DAEMON_IDLE);
> +		}
> +	}
>  
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=shutdown");
> -#endif
> -	running_state = DAEMON_SHUTDOWN;
>  	lock(vecs->lock);
>  	if (conf->queue_without_daemon == QUE_NO_DAEMON_OFF)
>  		vector_foreach_slot(vecs->mpvec, mpp, i)
> @@ -2253,7 +2325,6 @@ main (int argc, char *argv[])
>  	int foreground = 0;
>  
>  	logsink = 1;
> -	running_state = DAEMON_INIT;
>  	dm_init();
>  
>  	if (getuid() != 0) {
> diff --git a/multipathd/main.h b/multipathd/main.h
> index d1a6d71..10b3a6d 100644
> --- a/multipathd/main.h
> +++ b/multipathd/main.h
> @@ -7,6 +7,7 @@ enum daemon_status {
>      DAEMON_INIT,
>      DAEMON_START,
>      DAEMON_CONFIGURE,
> +    DAEMON_IDLE,
>      DAEMON_RUNNING,
>      DAEMON_SHUTDOWN,
>  };
> @@ -26,6 +27,7 @@ int ev_remove_path (struct path *, struct vectors *);
>  int ev_add_map (char *, char *, struct vectors *);
>  int ev_remove_map (char *, char *, int, struct vectors *);
>  void sync_map_state (struct multipath *);
> +int set_config_state(enum daemon_status);
>  void * mpath_alloc_prin_response(int prin_sa);
>  int prin_do_scsi_ioctl(char *, int rq_servact, struct prin_resp * resp,
>         int noisy);
> -- 
> 2.6.6

Re: [PATCH 55/57] multipathd: asynchronous configuration

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:56PM +0200, Hannes Reinecke wrote:

Oh, I did have one other thought.

If you're not changing the signal mask in uevent_listen, can't you just
use poll?  Otherwise people are going to try and figure out what's going
on with the signals that they're missing (at least that's what I did).


> For initial configuration multipathd waits until it has synchronized
> with the existing setup. On larger systems this takes up quite
> some time (I've measured 80 seconds on a system with 1024 paths)
> causing systemd to stall and the system to fail booting.
> This patch makes the initial configuration asynchronous, and
> using the same codepath as the existing 'reconfigure' CLI
> command.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  libmultipath/uevent.c     |  10 +--
>  multipathd/cli_handlers.c |  14 ++-
>  multipathd/main.c         | 219 ++++++++++++++++++++++++++++++----------------
>  multipathd/main.h         |   2 +
>  4 files changed, 157 insertions(+), 88 deletions(-)
> 
> diff --git a/libmultipath/uevent.c b/libmultipath/uevent.c
> index 478c6ce..fbe9c44 100644
> --- a/libmultipath/uevent.c
> +++ b/libmultipath/uevent.c
> @@ -529,8 +529,6 @@ int uevent_listen(struct udev *udev)
>  	}
>  
>  	pthread_sigmask(SIG_SETMASK, NULL, &mask);
> -	sigdelset(&mask, SIGHUP);
> -	sigdelset(&mask, SIGUSR1);
>  	events = 0;
>  	while (1) {
>  		struct uevent *uev;
> @@ -561,9 +559,11 @@ int uevent_listen(struct udev *udev)
>  			continue;
>  		}
>  		if (fdcount < 0) {
> -			if (errno != EINTR)
> -				condlog(0, "error receiving "
> -					"uevent message: %m");
> +			if (errno == EINTR)
> +				continue;
> +
> +			condlog(0, "error receiving "
> +				"uevent message: %m");
>  			err = -errno;
>  			break;
>  		}
> diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
> index dbdcbc2..0397353 100644
> --- a/multipathd/cli_handlers.c
> +++ b/multipathd/cli_handlers.c
> @@ -909,17 +909,13 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)
>  int
>  cli_reconfigure(void * v, char ** reply, int * len, void * data)
>  {
> -	struct vectors * vecs = (struct vectors *)data;
> -
> -	if (need_to_delay_reconfig(vecs)) {
> -		conf->delayed_reconfig = 1;
> -		condlog(2, "delaying reconfigure (operator)");
> -		return 0;
> -	}
> -
>  	condlog(2, "reconfigure (operator)");
>  
> -	return reconfigure(vecs);
> +	if (set_config_state(DAEMON_CONFIGURE) == ETIMEDOUT) {
> +		condlog(2, "timeout starting reconfiguration");
> +		return 1;
> +	}
> +	return 0;
>  }
>  
>  int
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 77eb498..41b5a49 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -97,10 +97,11 @@ struct mpath_event_param
>  unsigned int mpath_mx_alloc_len;
>  
>  int logsink;
> -enum daemon_status running_state;
> +enum daemon_status running_state = DAEMON_INIT;
>  pid_t daemon_pid;
> +pthread_mutex_t config_lock = PTHREAD_MUTEX_INITIALIZER;
> +pthread_cond_t config_cond = PTHREAD_COND_INITIALIZER;
>  
> -static sem_t exit_sem;
>  /*
>   * global copy of vecs for use in sig handlers
>   */
> @@ -108,6 +109,94 @@ struct vectors * gvecs;
>  
>  struct udev * udev;
>  
> +const char *
> +daemon_status(void)
> +{
> +	switch (running_state) {
> +	case DAEMON_INIT:
> +		return "init";
> +	case DAEMON_START:
> +		return "startup";
> +	case DAEMON_CONFIGURE:
> +		return "configure";
> +	case DAEMON_IDLE:
> +		return "idle";
> +	case DAEMON_RUNNING:
> +		return "running";
> +	case DAEMON_SHUTDOWN:
> +		return "shutdown";
> +	}
> +	return NULL;
> +}
> +
> +/*
> + * I love you too, systemd ...
> + */
> +const char *
> +sd_notify_status(void)
> +{
> +	switch (running_state) {
> +	case DAEMON_INIT:
> +		return "STATUS=init";
> +	case DAEMON_START:
> +		return "STATUS=startup";
> +	case DAEMON_CONFIGURE:
> +		return "STATUS=configure";
> +	case DAEMON_IDLE:
> +		return "STATUS=idle";
> +	case DAEMON_RUNNING:
> +		return "STATUS=running";
> +	case DAEMON_SHUTDOWN:
> +		return "STATUS=shutdown";
> +	}
> +	return NULL;
> +}
> +
> +static void config_cleanup(void *arg)
> +{
> +	pthread_mutex_unlock(&config_lock);
> +}
> +
> +void post_config_state(enum daemon_status state)
> +{
> +	pthread_mutex_lock(&config_lock);
> +	if (state != running_state) {
> +		running_state = state;
> +		pthread_cond_broadcast(&config_cond);
> +#ifdef USE_SYSTEMD
> +		sd_notify(0, sd_notify_status());
> +#endif
> +	}
> +	pthread_mutex_unlock(&config_lock);
> +}
> +
> +int set_config_state(enum daemon_status state)
> +{
> +	int rc = 0;
> +
> +	pthread_cleanup_push(config_cleanup, NULL);
> +	pthread_mutex_lock(&config_lock);
> +	if (running_state != state) {
> +		if (running_state != DAEMON_IDLE) {
> +			struct timespec ts;
> +
> +			clock_gettime(CLOCK_REALTIME, &ts);
> +			ts.tv_sec += 1;
> +			rc = pthread_cond_timedwait(&config_cond,
> +						    &config_lock, &ts);
> +		}
> +		if (!rc) {
> +			running_state = state;
> +			pthread_cond_broadcast(&config_cond);
> +#ifdef USE_SYSTEMD
> +			sd_notify(0, sd_notify_status());
> +#endif
> +		}
> +	}
> +	pthread_cleanup_pop(1);
> +	return rc;
> +}
> +
>  static int
>  need_switch_pathgroup (struct multipath * mpp, int refresh)
>  {
> @@ -352,7 +441,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
>  	if (mpp) {
>  		if (mpp->wait_for_udev > 1) {
>  			if (update_map(mpp, vecs))
> -			/* setup multipathd removed the map */
> +				/* setup multipathd removed the map */
>  				return 1;
>  		}
>  		if (mpp->wait_for_udev) {
> @@ -360,7 +449,7 @@ ev_add_map (char * dev, char * alias, struct vectors * vecs)
>  			if (conf->delayed_reconfig &&
>  			    !need_to_delay_reconfig(vecs)) {
>  				condlog(2, "reconfigure (delayed)");
> -				reconfigure(vecs);
> +				set_config_state(DAEMON_CONFIGURE);
>  				return 0;
>  			}
>  		}
> @@ -903,6 +992,16 @@ uev_trigger (struct uevent * uev, void * trigger_data)
>  	if (uev_discard(uev->devpath))
>  		return 0;
>  
> +	pthread_cleanup_push(config_cleanup, NULL);
> +	pthread_mutex_lock(&config_lock);
> +	if (running_state != DAEMON_IDLE &&
> +	    running_state != DAEMON_RUNNING)
> +		pthread_cond_wait(&config_cond, &config_lock);
> +	pthread_cleanup_pop(1);
> +
> +	if (running_state == DAEMON_SHUTDOWN)
> +		return 0;
> +
>  	pthread_cleanup_push(cleanup_lock, &vecs->lock);
>  	lock(vecs->lock);
>  	pthread_testcancel();
> @@ -1031,25 +1130,7 @@ uxlsnrloop (void * ap)
>  void
>  exit_daemon (void)
>  {
> -	sem_post(&exit_sem);
> -}
> -
> -const char *
> -daemon_status(void)
> -{
> -	switch (running_state) {
> -	case DAEMON_INIT:
> -		return "init";
> -	case DAEMON_START:
> -		return "startup";
> -	case DAEMON_CONFIGURE:
> -		return "configure";
> -	case DAEMON_RUNNING:
> -		return "running";
> -	case DAEMON_SHUTDOWN:
> -		return "shutdown";
> -	}
> -	return NULL;
> +	post_config_state(DAEMON_SHUTDOWN);
>  }
>  
>  static void
> @@ -1178,7 +1259,7 @@ missing_uev_wait_tick(struct vectors *vecs)
>  	if (timed_out && conf->delayed_reconfig &&
>  	    !need_to_delay_reconfig(vecs)) {
>  		condlog(2, "reconfigure (delayed)");
> -		reconfigure(vecs);
> +		set_config_state(DAEMON_CONFIGURE);
>  	}
>  }
>  
> @@ -1541,11 +1622,18 @@ checkerloop (void *ap)
>  
>  	while (1) {
>  		struct timeval diff_time, start_time, end_time;
> -		int num_paths = 0, ticks = 0, signo, strict_timing;
> +		int num_paths = 0, ticks = 0, signo, strict_timing, rc = 0;
>  		sigset_t mask;
>  
>  		if (gettimeofday(&start_time, NULL) != 0)
>  			start_time.tv_sec = 0;
> +
> +		rc = set_config_state(DAEMON_RUNNING);
> +		if (rc == ETIMEDOUT) {
> +			condlog(4, "timeout waiting for DAEMON_IDLE");
> +			continue;
> +		}
> +
>  		pthread_cleanup_push(cleanup_lock, &vecs->lock);
>  		lock(vecs->lock);
>  		pthread_testcancel();
> @@ -1600,6 +1688,7 @@ checkerloop (void *ap)
>  			}
>  		}
>  
> +		post_config_state(DAEMON_IDLE);
>  		if (!strict_timing)
>  			sleep(1);
>  		else {
> @@ -1734,8 +1823,6 @@ reconfigure (struct vectors * vecs)
>  	struct config * old = conf;
>  	int retval = 1;
>  
> -	running_state = DAEMON_CONFIGURE;
> -
>  	/*
>  	 * free old map and path vectors ... they use old conf state
>  	 */
> @@ -1765,8 +1852,6 @@ reconfigure (struct vectors * vecs)
>  	}
>  	uxsock_timeout = conf->uxsock_timeout;
>  
> -	running_state = DAEMON_RUNNING;
> -
>  	return retval;
>  }
>  
> @@ -1819,20 +1904,9 @@ signal_set(int signo, void (*func) (int))
>  void
>  handle_signals(void)
>  {
> -	if (reconfig_sig && running_state == DAEMON_RUNNING) {
> -		pthread_cleanup_push(cleanup_lock,
> -				&gvecs->lock);
> -		lock(gvecs->lock);
> -		pthread_testcancel();
> -		if (need_to_delay_reconfig(gvecs)) {
> -			conf->delayed_reconfig = 1;
> -			condlog(2, "delaying reconfigure (signal)");
> -		}
> -		else {
> -			condlog(2, "reconfigure (signal)");
> -			reconfigure(gvecs);
> -		}
> -		lock_cleanup_pop(gvecs->lock);
> +	if (reconfig_sig) {
> +		condlog(2, "reconfigure (signal)");
> +		set_config_state(DAEMON_CONFIGURE);
>  	}
>  	if (log_reset_sig) {
>  		condlog(2, "reset log (signal)");
> @@ -1966,7 +2040,6 @@ child (void * param)
>  	char *envp;
>  
>  	mlockall(MCL_CURRENT | MCL_FUTURE);
> -	sem_init(&exit_sem, 0, 0);
>  	signal_init();
>  
>  	udev = udev_new();
> @@ -1987,11 +2060,8 @@ child (void * param)
>  		exit(1);
>  	}
>  
> -	running_state = DAEMON_START;
> +	post_config_state(DAEMON_START);
>  
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=startup");
> -#endif
>  	condlog(2, "--------start up--------");
>  	condlog(2, "read " DEFAULT_CONFIGFILE);
>  
> @@ -2067,6 +2137,11 @@ child (void * param)
>  	}
>  #endif
>  	/*
> +	 * Signal start of configuration
> +	 */
> +	post_config_state(DAEMON_CONFIGURE);
> +
> +	/*
>  	 * Start uevent listener early to catch events
>  	 */
>  	if ((rc = pthread_create(&uevent_thr, &uevent_attr, ueventloop, udev))) {
> @@ -2078,21 +2153,6 @@ child (void * param)
>  		condlog(0, "failed to create cli listener: %d", rc);
>  		goto failed;
>  	}
> -	/*
> -	 * fetch and configure both paths and multipaths
> -	 */
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=configure");
> -#endif
> -	running_state = DAEMON_CONFIGURE;
> -
> -	lock(vecs->lock);
> -	if (configure(vecs, 1)) {
> -		unlock(vecs->lock);
> -		condlog(0, "failure during configuration");
> -		goto failed;
> -	}
> -	unlock(vecs->lock);
>  
>  	/*
>  	 * start threads
> @@ -2107,20 +2167,32 @@ child (void * param)
>  	}
>  	pthread_attr_destroy(&misc_attr);
>  
> -	running_state = DAEMON_RUNNING;
>  #ifdef USE_SYSTEMD
> -	sd_notify(0, "READY=1\nSTATUS=running");
> +	sd_notify(0, "READY=1");
>  #endif
>  
> -	/*
> -	 * exit path
> -	 */
> -	while(sem_wait(&exit_sem) != 0); /* Do nothing */
> +	while (running_state != DAEMON_SHUTDOWN) {
> +		pthread_cleanup_push(config_cleanup, NULL);
> +		pthread_mutex_lock(&config_lock);
> +		if (running_state != DAEMON_CONFIGURE &&
> +		    running_state != DAEMON_SHUTDOWN) {
> +			pthread_cond_wait(&config_cond, &config_lock);
> +		}
> +		pthread_cleanup_pop(1);
> +		if (running_state == DAEMON_CONFIGURE) {
> +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +			lock(vecs->lock);
> +			pthread_testcancel();
> +			if (!need_to_delay_reconfig(vecs)) {
> +				reconfigure(vecs);
> +			} else {
> +				conf->delayed_reconfig = 1;
> +			}
> +			lock_cleanup_pop(vecs->lock);
> +			post_config_state(DAEMON_IDLE);
> +		}
> +	}
>  
> -#ifdef USE_SYSTEMD
> -	sd_notify(0, "STATUS=shutdown");
> -#endif
> -	running_state = DAEMON_SHUTDOWN;
>  	lock(vecs->lock);
>  	if (conf->queue_without_daemon == QUE_NO_DAEMON_OFF)
>  		vector_foreach_slot(vecs->mpvec, mpp, i)
> @@ -2253,7 +2325,6 @@ main (int argc, char *argv[])
>  	int foreground = 0;
>  
>  	logsink = 1;
> -	running_state = DAEMON_INIT;
>  	dm_init();
>  
>  	if (getuid() != 0) {
> diff --git a/multipathd/main.h b/multipathd/main.h
> index d1a6d71..10b3a6d 100644
> --- a/multipathd/main.h
> +++ b/multipathd/main.h
> @@ -7,6 +7,7 @@ enum daemon_status {
>      DAEMON_INIT,
>      DAEMON_START,
>      DAEMON_CONFIGURE,
> +    DAEMON_IDLE,
>      DAEMON_RUNNING,
>      DAEMON_SHUTDOWN,
>  };
> @@ -26,6 +27,7 @@ int ev_remove_path (struct path *, struct vectors *);
>  int ev_add_map (char *, char *, struct vectors *);
>  int ev_remove_map (char *, char *, int, struct vectors *);
>  void sync_map_state (struct multipath *);
> +int set_config_state(enum daemon_status);
>  void * mpath_alloc_prin_response(int prin_sa);
>  int prin_do_scsi_ioctl(char *, int rq_servact, struct prin_resp * resp,
>         int noisy);
> -- 
> 2.6.6

Re: [PATCH 55/57] multipathd: asynchronous configuration

[Hannes Reinecke]

On 05/03/2016 08:25 PM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:56PM +0200, Hannes Reinecke wrote:
> 
> Oh, I did have one other thought.
> 
> If you're not changing the signal mask in uevent_listen, can't you just
> use poll?  Otherwise people are going to try and figure out what's going
> on with the signals that they're missing (at least that's what I did).
> 
Sure. Don't have any issue with that.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[PATCH 56/57] multipathd: push down lock in checkerloop()

[Hannes Reinecke]

Instead of grabbing the lock at the start of the checkerloop
and releasing it at the end we should be holding it only
during the time when we actually need it.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/main.c | 136 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 87 insertions(+), 49 deletions(-)

diff --git a/multipathd/main.c b/multipathd/main.c
index 41b5a49..132101f 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -416,7 +416,11 @@ uev_add_map (struct uevent * uev, struct vectors * vecs)
 			return 1;
 		}
 	}
+	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+	lock(vecs->lock);
+	pthread_testcancel();
 	rc = ev_add_map(uev->kernel, alias, vecs);
+	lock_cleanup_pop(vecs->lock);
 	FREE(alias);
 	return rc;
 }
@@ -512,6 +516,10 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
 		return 0;
 	}
 	minor = uevent_get_minor(uev);
+
+	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+	lock(vecs->lock);
+	pthread_testcancel();
 	mpp = find_mp_by_minor(vecs->mpvec, minor);
 
 	if (!mpp) {
@@ -528,10 +536,12 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
 	orphan_paths(vecs->pathvec, mpp);
 	remove_map_and_stop_waiter(mpp, vecs, 1);
 out:
+	lock_cleanup_pop(vecs->lock);
 	FREE(alias);
 	return 0;
 }
 
+/* Called from CLI handler */
 int
 ev_remove_map (char * devname, char * alias, int minor, struct vectors * vecs)
 {
@@ -567,6 +577,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
 		return 1;
 	}
 
+	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+	lock(vecs->lock);
+	pthread_testcancel();
 	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
 	if (pp) {
 		int r;
@@ -594,8 +607,10 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
 				ret = 1;
 			}
 		}
-		return ret;
 	}
+	lock_cleanup_pop(vecs->lock);
+	if (pp)
+		return ret;
 
 	/*
 	 * get path vital state
@@ -608,6 +623,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
 		condlog(3, "%s: failed to get path info", uev->kernel);
 		return 1;
 	}
+	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+	lock(vecs->lock);
+	pthread_testcancel();
 	ret = store_path(vecs->pathvec, pp);
 	if (!ret) {
 		pp->checkint = conf->checkint;
@@ -619,7 +637,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
 		free_path(pp);
 		ret = 1;
 	}
-
+	lock_cleanup_pop(vecs->lock);
 	return ret;
 }
 
@@ -687,12 +705,12 @@ rescan:
 			 */
 			start_waiter = 1;
 		}
-		else
+		if (!start_waiter)
 			goto fail; /* leave path added to pathvec */
 	}
 
-	/* persistent reseravtion check*/
-	mpath_pr_event_handle(pp);	
+	/* persistent reservation check*/
+	mpath_pr_event_handle(pp);
 
 	/*
 	 * push the map to the device-mapper
@@ -720,7 +738,7 @@ retry:
 		 * deal with asynchronous uevents :((
 		 */
 		if (mpp->action == ACT_RELOAD && retries-- > 0) {
-			condlog(0, "%s: uev_add_path sleep", mpp->alias);
+			condlog(0, "%s: ev_add_path sleep", mpp->alias);
 			sleep(1);
 			update_mpp_paths(mpp, vecs->pathvec);
 			goto rescan;
@@ -749,8 +767,7 @@ retry:
 		condlog(2, "%s [%s]: path added to devmap %s",
 			pp->dev, pp->dev_t, mpp->alias);
 		return 0;
-	}
-	else
+	} else
 		goto fail;
 
 fail_map:
@@ -764,17 +781,22 @@ static int
 uev_remove_path (struct uevent *uev, struct vectors * vecs)
 {
 	struct path *pp;
+	int ret;
 
 	condlog(2, "%s: remove path (uevent)", uev->kernel);
+	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+	lock(vecs->lock);
+	pthread_testcancel();
 	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
-
+	if (pp)
+		ret = ev_remove_path(pp, vecs);
+	lock_cleanup_pop(vecs->lock);
 	if (!pp) {
 		/* Not an error; path might have been purged earlier */
 		condlog(0, "%s: path already removed", uev->kernel);
 		return 0;
 	}
-
-	return ev_remove_path(pp, vecs);
+	return ret;
 }
 
 int
@@ -877,35 +899,50 @@ static int
 uev_update_path (struct uevent *uev, struct vectors * vecs)
 {
 	int ro, retval = 0;
-	struct path * pp;
-
-	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
-	if (!pp) {
-		condlog(0, "%s: spurious uevent, path not found",
-			uev->kernel);
-		return 1;
-	}
-
-	if (pp->initialized == INIT_REQUESTED_UDEV)
-		return uev_add_path(uev, vecs);
 
 	ro = uevent_get_disk_ro(uev);
 
 	if (ro >= 0) {
+		struct path * pp;
+		struct multipath *mpp = NULL;
+
 		condlog(2, "%s: update path write_protect to '%d' (uevent)",
 			uev->kernel, ro);
-		if (pp->mpp) {
-			if (pp->mpp->wait_for_udev) {
-				pp->mpp->wait_for_udev = 2;
-				return 0;
+		pthread_cleanup_push(cleanup_lock, &vecs->lock);
+		lock(vecs->lock);
+		pthread_testcancel();
+		/*
+		 * pthread_mutex_lock() and pthread_mutex_unlock()
+		 * need to be at the same indentation level, hence
+		 * this slightly convoluted codepath.
+		 */
+		pp = find_path_by_dev(vecs->pathvec, uev->kernel);
+		if (pp) {
+			if (pp->initialized == INIT_REQUESTED_UDEV) {
+				retval = 2;
+			} else {
+				mpp = pp->mpp;
+				if (mpp && mpp->wait_for_udev) {
+					mpp->wait_for_udev = 2;
+					mpp = NULL;
+					retval = 0;
+				}
 			}
+			if (mpp) {
+				retval = reload_map(vecs, mpp, 0);
 
-			retval = reload_map(vecs, pp->mpp, 0);
-
-			condlog(2, "%s: map %s reloaded (retval %d)",
-				uev->kernel, pp->mpp->alias, retval);
+				condlog(2, "%s: map %s reloaded (retval %d)",
+					uev->kernel, mpp->alias, retval);
+			}
 		}
-
+		lock_cleanup_pop(vecs->lock);
+		if (!pp) {
+			condlog(0, "%s: spurious uevent, path not found",
+				uev->kernel);
+			return 1;
+		}
+		if (retval == 2)
+			return uev_add_path(uev, vecs);
 	}
 
 	return retval;
@@ -1002,10 +1039,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
 	if (running_state == DAEMON_SHUTDOWN)
 		return 0;
 
-	pthread_cleanup_push(cleanup_lock, &vecs->lock);
-	lock(vecs->lock);
-	pthread_testcancel();
-
 	/*
 	 * device map event
 	 * Add events are ignored here as the tables
@@ -1044,7 +1077,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
 	}
 
 out:
-	lock_cleanup_pop(vecs->lock);
 	return r;
 }
 
@@ -1627,17 +1659,6 @@ checkerloop (void *ap)
 
 		if (gettimeofday(&start_time, NULL) != 0)
 			start_time.tv_sec = 0;
-
-		rc = set_config_state(DAEMON_RUNNING);
-		if (rc == ETIMEDOUT) {
-			condlog(4, "timeout waiting for DAEMON_IDLE");
-			continue;
-		}
-
-		pthread_cleanup_push(cleanup_lock, &vecs->lock);
-		lock(vecs->lock);
-		pthread_testcancel();
-		strict_timing = conf->strict_timing;
 		if (start_time.tv_sec && last_time.tv_sec) {
 			timersub(&start_time, &last_time, &diff_time);
 			condlog(4, "tick (%lu.%06lu secs)",
@@ -1653,28 +1674,45 @@ checkerloop (void *ap)
 		if (use_watchdog)
 			sd_notify(0, "WATCHDOG=1");
 #endif
+		rc = set_config_state(DAEMON_RUNNING);
+		if (rc == ETIMEDOUT) {
+			condlog(4, "timeout waiting for DAEMON_IDLE");
+			continue;
+		}
+		strict_timing = conf->strict_timing;
 		if (vecs->pathvec) {
+			pthread_cleanup_push(cleanup_lock, &vecs->lock);
+			lock(vecs->lock);
+			pthread_testcancel();
 			vector_foreach_slot (vecs->pathvec, pp, i) {
 				num_paths += check_path(vecs, pp, ticks);
 			}
+			lock_cleanup_pop(vecs->lock);
 		}
 		if (vecs->mpvec) {
+			pthread_cleanup_push(cleanup_lock, &vecs->lock);
+			lock(vecs->lock);
+			pthread_testcancel();
 			defered_failback_tick(vecs->mpvec);
 			retry_count_tick(vecs->mpvec);
 			missing_uev_wait_tick(vecs);
+			lock_cleanup_pop(vecs->lock);
 		}
 		if (count)
 			count--;
 		else {
+			pthread_cleanup_push(cleanup_lock, &vecs->lock);
+			lock(vecs->lock);
+			pthread_testcancel();
 			condlog(4, "map garbage collection");
 			mpvec_garbage_collector(vecs);
 			count = MAPGCINT;
+			lock_cleanup_pop(vecs->lock);
 		}
 
-		lock_cleanup_pop(vecs->lock);
 		diff_time.tv_usec = 0;
 		if (start_time.tv_sec &&
-		    gettimeofday(&end_time, NULL)) {
+		    gettimeofday(&end_time, NULL) == 0) {
 			timersub(&end_time, &start_time, &diff_time);
 			if (num_paths) {
 				condlog(3, "checked %d path%s in %lu.%06lu secs",
-- 
2.6.6

Re: [PATCH 56/57] multipathd: push down lock in checkerloop()

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:57PM +0200, Hannes Reinecke wrote:
> Instead of grabbing the lock at the start of the checkerloop
> and releasing it at the end we should be holding it only
> during the time when we actually need it.

I'm pretty sure that this can cause crashes if multipathd reconfigures
when it's just started servicing a uevent. It goes like this. The
uevq_thr thread calls uev_trigger(), which checks the running_state and
sees that it's DAEMON_IDLE.  The uxlsnr_thr thread gets a
reconfiguration request, which eventually calls set_config_state().
set_config_state() sees the state is DAEMON_IDLE, sets it to
DAEMON_CONFIGURE, and returns without waiting.  This wakes up the child
thread, which runs reconfigure().  reconfigure() sets conf to NULL. Then
the uevq_thr thread in uev_trigger() calls filter_devnode()
dereferencing conf->blist_devnode and conf->elist_devnode. conf is NULL,
and thus it crashes.

The short version is that multipathd has been using the vecs lock to
protect conf.  With your change to uev_trigger, you access conf outside
of the vecs lock. This isn't a problem in checkerloop(), since you can't
reconfigure while multipathd is in the DAEMON_RUNNING state, so that is
protecting conf. You need to do the same
set_config_state(DAEMON_RUNNING) in uev_trigger to make this safe.

-Ben

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  multipathd/main.c | 136 ++++++++++++++++++++++++++++++++++--------------------
>  1 file changed, 87 insertions(+), 49 deletions(-)
> 
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 41b5a49..132101f 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -416,7 +416,11 @@ uev_add_map (struct uevent * uev, struct vectors * vecs)
>  			return 1;
>  		}
>  	}
> +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +	lock(vecs->lock);
> +	pthread_testcancel();
>  	rc = ev_add_map(uev->kernel, alias, vecs);
> +	lock_cleanup_pop(vecs->lock);
>  	FREE(alias);
>  	return rc;
>  }
> @@ -512,6 +516,10 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
>  		return 0;
>  	}
>  	minor = uevent_get_minor(uev);
> +
> +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +	lock(vecs->lock);
> +	pthread_testcancel();
>  	mpp = find_mp_by_minor(vecs->mpvec, minor);
>  
>  	if (!mpp) {
> @@ -528,10 +536,12 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
>  	orphan_paths(vecs->pathvec, mpp);
>  	remove_map_and_stop_waiter(mpp, vecs, 1);
>  out:
> +	lock_cleanup_pop(vecs->lock);
>  	FREE(alias);
>  	return 0;
>  }
>  
> +/* Called from CLI handler */
>  int
>  ev_remove_map (char * devname, char * alias, int minor, struct vectors * vecs)
>  {
> @@ -567,6 +577,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
>  		return 1;
>  	}
>  
> +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +	lock(vecs->lock);
> +	pthread_testcancel();
>  	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
>  	if (pp) {
>  		int r;
> @@ -594,8 +607,10 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
>  				ret = 1;
>  			}
>  		}
> -		return ret;
>  	}
> +	lock_cleanup_pop(vecs->lock);
> +	if (pp)
> +		return ret;
>  
>  	/*
>  	 * get path vital state
> @@ -608,6 +623,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
>  		condlog(3, "%s: failed to get path info", uev->kernel);
>  		return 1;
>  	}
> +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +	lock(vecs->lock);
> +	pthread_testcancel();
>  	ret = store_path(vecs->pathvec, pp);
>  	if (!ret) {
>  		pp->checkint = conf->checkint;
> @@ -619,7 +637,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
>  		free_path(pp);
>  		ret = 1;
>  	}
> -
> +	lock_cleanup_pop(vecs->lock);
>  	return ret;
>  }
>  
> @@ -687,12 +705,12 @@ rescan:
>  			 */
>  			start_waiter = 1;
>  		}
> -		else
> +		if (!start_waiter)
>  			goto fail; /* leave path added to pathvec */
>  	}
>  
> -	/* persistent reseravtion check*/
> -	mpath_pr_event_handle(pp);	
> +	/* persistent reservation check*/
> +	mpath_pr_event_handle(pp);
>  
>  	/*
>  	 * push the map to the device-mapper
> @@ -720,7 +738,7 @@ retry:
>  		 * deal with asynchronous uevents :((
>  		 */
>  		if (mpp->action == ACT_RELOAD && retries-- > 0) {
> -			condlog(0, "%s: uev_add_path sleep", mpp->alias);
> +			condlog(0, "%s: ev_add_path sleep", mpp->alias);
>  			sleep(1);
>  			update_mpp_paths(mpp, vecs->pathvec);
>  			goto rescan;
> @@ -749,8 +767,7 @@ retry:
>  		condlog(2, "%s [%s]: path added to devmap %s",
>  			pp->dev, pp->dev_t, mpp->alias);
>  		return 0;
> -	}
> -	else
> +	} else
>  		goto fail;
>  
>  fail_map:
> @@ -764,17 +781,22 @@ static int
>  uev_remove_path (struct uevent *uev, struct vectors * vecs)
>  {
>  	struct path *pp;
> +	int ret;
>  
>  	condlog(2, "%s: remove path (uevent)", uev->kernel);
> +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +	lock(vecs->lock);
> +	pthread_testcancel();
>  	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> -
> +	if (pp)
> +		ret = ev_remove_path(pp, vecs);
> +	lock_cleanup_pop(vecs->lock);
>  	if (!pp) {
>  		/* Not an error; path might have been purged earlier */
>  		condlog(0, "%s: path already removed", uev->kernel);
>  		return 0;
>  	}
> -
> -	return ev_remove_path(pp, vecs);
> +	return ret;
>  }
>  
>  int
> @@ -877,35 +899,50 @@ static int
>  uev_update_path (struct uevent *uev, struct vectors * vecs)
>  {
>  	int ro, retval = 0;
> -	struct path * pp;
> -
> -	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> -	if (!pp) {
> -		condlog(0, "%s: spurious uevent, path not found",
> -			uev->kernel);
> -		return 1;
> -	}
> -
> -	if (pp->initialized == INIT_REQUESTED_UDEV)
> -		return uev_add_path(uev, vecs);
>  
>  	ro = uevent_get_disk_ro(uev);
>  
>  	if (ro >= 0) {
> +		struct path * pp;
> +		struct multipath *mpp = NULL;
> +
>  		condlog(2, "%s: update path write_protect to '%d' (uevent)",
>  			uev->kernel, ro);
> -		if (pp->mpp) {
> -			if (pp->mpp->wait_for_udev) {
> -				pp->mpp->wait_for_udev = 2;
> -				return 0;
> +		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +		lock(vecs->lock);
> +		pthread_testcancel();
> +		/*
> +		 * pthread_mutex_lock() and pthread_mutex_unlock()
> +		 * need to be at the same indentation level, hence
> +		 * this slightly convoluted codepath.
> +		 */
> +		pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> +		if (pp) {
> +			if (pp->initialized == INIT_REQUESTED_UDEV) {
> +				retval = 2;
> +			} else {
> +				mpp = pp->mpp;
> +				if (mpp && mpp->wait_for_udev) {
> +					mpp->wait_for_udev = 2;
> +					mpp = NULL;
> +					retval = 0;
> +				}
>  			}
> +			if (mpp) {
> +				retval = reload_map(vecs, mpp, 0);
>  
> -			retval = reload_map(vecs, pp->mpp, 0);
> -
> -			condlog(2, "%s: map %s reloaded (retval %d)",
> -				uev->kernel, pp->mpp->alias, retval);
> +				condlog(2, "%s: map %s reloaded (retval %d)",
> +					uev->kernel, mpp->alias, retval);
> +			}
>  		}
> -
> +		lock_cleanup_pop(vecs->lock);
> +		if (!pp) {
> +			condlog(0, "%s: spurious uevent, path not found",
> +				uev->kernel);
> +			return 1;
> +		}
> +		if (retval == 2)
> +			return uev_add_path(uev, vecs);
>  	}
>  
>  	return retval;
> @@ -1002,10 +1039,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
>  	if (running_state == DAEMON_SHUTDOWN)
>  		return 0;
>  
> -	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> -	lock(vecs->lock);
> -	pthread_testcancel();
> -
>  	/*
>  	 * device map event
>  	 * Add events are ignored here as the tables
> @@ -1044,7 +1077,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
>  	}
>  
>  out:
> -	lock_cleanup_pop(vecs->lock);
>  	return r;
>  }
>  
> @@ -1627,17 +1659,6 @@ checkerloop (void *ap)
>  
>  		if (gettimeofday(&start_time, NULL) != 0)
>  			start_time.tv_sec = 0;
> -
> -		rc = set_config_state(DAEMON_RUNNING);
> -		if (rc == ETIMEDOUT) {
> -			condlog(4, "timeout waiting for DAEMON_IDLE");
> -			continue;
> -		}
> -
> -		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> -		lock(vecs->lock);
> -		pthread_testcancel();
> -		strict_timing = conf->strict_timing;
>  		if (start_time.tv_sec && last_time.tv_sec) {
>  			timersub(&start_time, &last_time, &diff_time);
>  			condlog(4, "tick (%lu.%06lu secs)",
> @@ -1653,28 +1674,45 @@ checkerloop (void *ap)
>  		if (use_watchdog)
>  			sd_notify(0, "WATCHDOG=1");
>  #endif
> +		rc = set_config_state(DAEMON_RUNNING);
> +		if (rc == ETIMEDOUT) {
> +			condlog(4, "timeout waiting for DAEMON_IDLE");
> +			continue;
> +		}
> +		strict_timing = conf->strict_timing;
>  		if (vecs->pathvec) {
> +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +			lock(vecs->lock);
> +			pthread_testcancel();
>  			vector_foreach_slot (vecs->pathvec, pp, i) {
>  				num_paths += check_path(vecs, pp, ticks);
>  			}
> +			lock_cleanup_pop(vecs->lock);
>  		}
>  		if (vecs->mpvec) {
> +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +			lock(vecs->lock);
> +			pthread_testcancel();
>  			defered_failback_tick(vecs->mpvec);
>  			retry_count_tick(vecs->mpvec);
>  			missing_uev_wait_tick(vecs);
> +			lock_cleanup_pop(vecs->lock);
>  		}
>  		if (count)
>  			count--;
>  		else {
> +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +			lock(vecs->lock);
> +			pthread_testcancel();
>  			condlog(4, "map garbage collection");
>  			mpvec_garbage_collector(vecs);
>  			count = MAPGCINT;
> +			lock_cleanup_pop(vecs->lock);
>  		}
>  
> -		lock_cleanup_pop(vecs->lock);
>  		diff_time.tv_usec = 0;
>  		if (start_time.tv_sec &&
> -		    gettimeofday(&end_time, NULL)) {
> +		    gettimeofday(&end_time, NULL) == 0) {
>  			timersub(&end_time, &start_time, &diff_time);
>  			if (num_paths) {
>  				condlog(3, "checked %d path%s in %lu.%06lu secs",
> -- 
> 2.6.6

Re: [PATCH 56/57] multipathd: push down lock in checkerloop()

[Benjamin Marzinski]

On Tue, May 03, 2016 at 05:17:34PM -0500, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:57PM +0200, Hannes Reinecke wrote:
> > Instead of grabbing the lock at the start of the checkerloop
> > and releasing it at the end we should be holding it only
> > during the time when we actually need it.
> 
> I'm pretty sure that this can cause crashes if multipathd reconfigures
> when it's just started servicing a uevent. It goes like this. The
> uevq_thr thread calls uev_trigger(), which checks the running_state and
> sees that it's DAEMON_IDLE.  The uxlsnr_thr thread gets a
> reconfiguration request, which eventually calls set_config_state().
> set_config_state() sees the state is DAEMON_IDLE, sets it to
> DAEMON_CONFIGURE, and returns without waiting.  This wakes up the child
> thread, which runs reconfigure().  reconfigure() sets conf to NULL. Then
> the uevq_thr thread in uev_trigger() calls filter_devnode()
> dereferencing conf->blist_devnode and conf->elist_devnode. conf is NULL,
> and thus it crashes.
> 
> The short version is that multipathd has been using the vecs lock to
> protect conf.  With your change to uev_trigger, you access conf outside
> of the vecs lock. This isn't a problem in checkerloop(), since you can't
> reconfigure while multipathd is in the DAEMON_RUNNING state, so that is
> protecting conf. You need to do the same
> set_config_state(DAEMON_RUNNING) in uev_trigger to make this safe.

Or not. ev_add_map calls set_config_state(DAEMON_CONFIGURE), so this
would deadlock if you called  set_config_state(DAEMON_RUNNING) in
uev_trigger. At any rate, either reconfigures need to be blocked while
running uev_trigger, or we need some sort of reference counting on the
config structure so that we don't free it when we do a reconfigure,
until everyone has switched to the new structure.

-Ben
> 
> -Ben
> 
> > 
> > Signed-off-by: Hannes Reinecke <hare@suse.de>
> > ---
> >  multipathd/main.c | 136 ++++++++++++++++++++++++++++++++++--------------------
> >  1 file changed, 87 insertions(+), 49 deletions(-)
> > 
> > diff --git a/multipathd/main.c b/multipathd/main.c
> > index 41b5a49..132101f 100644
> > --- a/multipathd/main.c
> > +++ b/multipathd/main.c
> > @@ -416,7 +416,11 @@ uev_add_map (struct uevent * uev, struct vectors * vecs)
> >  			return 1;
> >  		}
> >  	}
> > +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +	lock(vecs->lock);
> > +	pthread_testcancel();
> >  	rc = ev_add_map(uev->kernel, alias, vecs);
> > +	lock_cleanup_pop(vecs->lock);
> >  	FREE(alias);
> >  	return rc;
> >  }
> > @@ -512,6 +516,10 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
> >  		return 0;
> >  	}
> >  	minor = uevent_get_minor(uev);
> > +
> > +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +	lock(vecs->lock);
> > +	pthread_testcancel();
> >  	mpp = find_mp_by_minor(vecs->mpvec, minor);
> >  
> >  	if (!mpp) {
> > @@ -528,10 +536,12 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)
> >  	orphan_paths(vecs->pathvec, mpp);
> >  	remove_map_and_stop_waiter(mpp, vecs, 1);
> >  out:
> > +	lock_cleanup_pop(vecs->lock);
> >  	FREE(alias);
> >  	return 0;
> >  }
> >  
> > +/* Called from CLI handler */
> >  int
> >  ev_remove_map (char * devname, char * alias, int minor, struct vectors * vecs)
> >  {
> > @@ -567,6 +577,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
> >  		return 1;
> >  	}
> >  
> > +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +	lock(vecs->lock);
> > +	pthread_testcancel();
> >  	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> >  	if (pp) {
> >  		int r;
> > @@ -594,8 +607,10 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
> >  				ret = 1;
> >  			}
> >  		}
> > -		return ret;
> >  	}
> > +	lock_cleanup_pop(vecs->lock);
> > +	if (pp)
> > +		return ret;
> >  
> >  	/*
> >  	 * get path vital state
> > @@ -608,6 +623,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
> >  		condlog(3, "%s: failed to get path info", uev->kernel);
> >  		return 1;
> >  	}
> > +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +	lock(vecs->lock);
> > +	pthread_testcancel();
> >  	ret = store_path(vecs->pathvec, pp);
> >  	if (!ret) {
> >  		pp->checkint = conf->checkint;
> > @@ -619,7 +637,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)
> >  		free_path(pp);
> >  		ret = 1;
> >  	}
> > -
> > +	lock_cleanup_pop(vecs->lock);
> >  	return ret;
> >  }
> >  
> > @@ -687,12 +705,12 @@ rescan:
> >  			 */
> >  			start_waiter = 1;
> >  		}
> > -		else
> > +		if (!start_waiter)
> >  			goto fail; /* leave path added to pathvec */
> >  	}
> >  
> > -	/* persistent reseravtion check*/
> > -	mpath_pr_event_handle(pp);	
> > +	/* persistent reservation check*/
> > +	mpath_pr_event_handle(pp);
> >  
> >  	/*
> >  	 * push the map to the device-mapper
> > @@ -720,7 +738,7 @@ retry:
> >  		 * deal with asynchronous uevents :((
> >  		 */
> >  		if (mpp->action == ACT_RELOAD && retries-- > 0) {
> > -			condlog(0, "%s: uev_add_path sleep", mpp->alias);
> > +			condlog(0, "%s: ev_add_path sleep", mpp->alias);
> >  			sleep(1);
> >  			update_mpp_paths(mpp, vecs->pathvec);
> >  			goto rescan;
> > @@ -749,8 +767,7 @@ retry:
> >  		condlog(2, "%s [%s]: path added to devmap %s",
> >  			pp->dev, pp->dev_t, mpp->alias);
> >  		return 0;
> > -	}
> > -	else
> > +	} else
> >  		goto fail;
> >  
> >  fail_map:
> > @@ -764,17 +781,22 @@ static int
> >  uev_remove_path (struct uevent *uev, struct vectors * vecs)
> >  {
> >  	struct path *pp;
> > +	int ret;
> >  
> >  	condlog(2, "%s: remove path (uevent)", uev->kernel);
> > +	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +	lock(vecs->lock);
> > +	pthread_testcancel();
> >  	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> > -
> > +	if (pp)
> > +		ret = ev_remove_path(pp, vecs);
> > +	lock_cleanup_pop(vecs->lock);
> >  	if (!pp) {
> >  		/* Not an error; path might have been purged earlier */
> >  		condlog(0, "%s: path already removed", uev->kernel);
> >  		return 0;
> >  	}
> > -
> > -	return ev_remove_path(pp, vecs);
> > +	return ret;
> >  }
> >  
> >  int
> > @@ -877,35 +899,50 @@ static int
> >  uev_update_path (struct uevent *uev, struct vectors * vecs)
> >  {
> >  	int ro, retval = 0;
> > -	struct path * pp;
> > -
> > -	pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> > -	if (!pp) {
> > -		condlog(0, "%s: spurious uevent, path not found",
> > -			uev->kernel);
> > -		return 1;
> > -	}
> > -
> > -	if (pp->initialized == INIT_REQUESTED_UDEV)
> > -		return uev_add_path(uev, vecs);
> >  
> >  	ro = uevent_get_disk_ro(uev);
> >  
> >  	if (ro >= 0) {
> > +		struct path * pp;
> > +		struct multipath *mpp = NULL;
> > +
> >  		condlog(2, "%s: update path write_protect to '%d' (uevent)",
> >  			uev->kernel, ro);
> > -		if (pp->mpp) {
> > -			if (pp->mpp->wait_for_udev) {
> > -				pp->mpp->wait_for_udev = 2;
> > -				return 0;
> > +		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +		lock(vecs->lock);
> > +		pthread_testcancel();
> > +		/*
> > +		 * pthread_mutex_lock() and pthread_mutex_unlock()
> > +		 * need to be at the same indentation level, hence
> > +		 * this slightly convoluted codepath.
> > +		 */
> > +		pp = find_path_by_dev(vecs->pathvec, uev->kernel);
> > +		if (pp) {
> > +			if (pp->initialized == INIT_REQUESTED_UDEV) {
> > +				retval = 2;
> > +			} else {
> > +				mpp = pp->mpp;
> > +				if (mpp && mpp->wait_for_udev) {
> > +					mpp->wait_for_udev = 2;
> > +					mpp = NULL;
> > +					retval = 0;
> > +				}
> >  			}
> > +			if (mpp) {
> > +				retval = reload_map(vecs, mpp, 0);
> >  
> > -			retval = reload_map(vecs, pp->mpp, 0);
> > -
> > -			condlog(2, "%s: map %s reloaded (retval %d)",
> > -				uev->kernel, pp->mpp->alias, retval);
> > +				condlog(2, "%s: map %s reloaded (retval %d)",
> > +					uev->kernel, mpp->alias, retval);
> > +			}
> >  		}
> > -
> > +		lock_cleanup_pop(vecs->lock);
> > +		if (!pp) {
> > +			condlog(0, "%s: spurious uevent, path not found",
> > +				uev->kernel);
> > +			return 1;
> > +		}
> > +		if (retval == 2)
> > +			return uev_add_path(uev, vecs);
> >  	}
> >  
> >  	return retval;
> > @@ -1002,10 +1039,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
> >  	if (running_state == DAEMON_SHUTDOWN)
> >  		return 0;
> >  
> > -	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > -	lock(vecs->lock);
> > -	pthread_testcancel();
> > -
> >  	/*
> >  	 * device map event
> >  	 * Add events are ignored here as the tables
> > @@ -1044,7 +1077,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)
> >  	}
> >  
> >  out:
> > -	lock_cleanup_pop(vecs->lock);
> >  	return r;
> >  }
> >  
> > @@ -1627,17 +1659,6 @@ checkerloop (void *ap)
> >  
> >  		if (gettimeofday(&start_time, NULL) != 0)
> >  			start_time.tv_sec = 0;
> > -
> > -		rc = set_config_state(DAEMON_RUNNING);
> > -		if (rc == ETIMEDOUT) {
> > -			condlog(4, "timeout waiting for DAEMON_IDLE");
> > -			continue;
> > -		}
> > -
> > -		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > -		lock(vecs->lock);
> > -		pthread_testcancel();
> > -		strict_timing = conf->strict_timing;
> >  		if (start_time.tv_sec && last_time.tv_sec) {
> >  			timersub(&start_time, &last_time, &diff_time);
> >  			condlog(4, "tick (%lu.%06lu secs)",
> > @@ -1653,28 +1674,45 @@ checkerloop (void *ap)
> >  		if (use_watchdog)
> >  			sd_notify(0, "WATCHDOG=1");
> >  #endif
> > +		rc = set_config_state(DAEMON_RUNNING);
> > +		if (rc == ETIMEDOUT) {
> > +			condlog(4, "timeout waiting for DAEMON_IDLE");
> > +			continue;
> > +		}
> > +		strict_timing = conf->strict_timing;
> >  		if (vecs->pathvec) {
> > +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +			lock(vecs->lock);
> > +			pthread_testcancel();
> >  			vector_foreach_slot (vecs->pathvec, pp, i) {
> >  				num_paths += check_path(vecs, pp, ticks);
> >  			}
> > +			lock_cleanup_pop(vecs->lock);
> >  		}
> >  		if (vecs->mpvec) {
> > +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +			lock(vecs->lock);
> > +			pthread_testcancel();
> >  			defered_failback_tick(vecs->mpvec);
> >  			retry_count_tick(vecs->mpvec);
> >  			missing_uev_wait_tick(vecs);
> > +			lock_cleanup_pop(vecs->lock);
> >  		}
> >  		if (count)
> >  			count--;
> >  		else {
> > +			pthread_cleanup_push(cleanup_lock, &vecs->lock);
> > +			lock(vecs->lock);
> > +			pthread_testcancel();
> >  			condlog(4, "map garbage collection");
> >  			mpvec_garbage_collector(vecs);
> >  			count = MAPGCINT;
> > +			lock_cleanup_pop(vecs->lock);
> >  		}
> >  
> > -		lock_cleanup_pop(vecs->lock);
> >  		diff_time.tv_usec = 0;
> >  		if (start_time.tv_sec &&
> > -		    gettimeofday(&end_time, NULL)) {
> > +		    gettimeofday(&end_time, NULL) == 0) {
> >  			timersub(&end_time, &start_time, &diff_time);
> >  			if (num_paths) {
> >  				condlog(3, "checked %d path%s in %lu.%06lu secs",
> > -- 
> > 2.6.6
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: [PATCH 56/57] multipathd: push down lock in checkerloop()

[Hannes Reinecke]

On 05/04/2016 01:24 AM, Benjamin Marzinski wrote:
> On Tue, May 03, 2016 at 05:17:34PM -0500, Benjamin Marzinski wrote:
>> On Wed, Apr 27, 2016 at 01:10:57PM +0200, Hannes Reinecke wrote:
>>> Instead of grabbing the lock at the start of the checkerloop
>>> and releasing it at the end we should be holding it only
>>> during the time when we actually need it.
>>
>> I'm pretty sure that this can cause crashes if multipathd reconfigures
>> when it's just started servicing a uevent. It goes like this. The
>> uevq_thr thread calls uev_trigger(), which checks the running_state and
>> sees that it's DAEMON_IDLE.  The uxlsnr_thr thread gets a
>> reconfiguration request, which eventually calls set_config_state().
>> set_config_state() sees the state is DAEMON_IDLE, sets it to
>> DAEMON_CONFIGURE, and returns without waiting.  This wakes up the child
>> thread, which runs reconfigure().  reconfigure() sets conf to NULL. Then
>> the uevq_thr thread in uev_trigger() calls filter_devnode()
>> dereferencing conf->blist_devnode and conf->elist_devnode. conf is NULL,
>> and thus it crashes.
>>
>> The short version is that multipathd has been using the vecs lock to
>> protect conf.  With your change to uev_trigger, you access conf outside
>> of the vecs lock. This isn't a problem in checkerloop(), since you can't
>> reconfigure while multipathd is in the DAEMON_RUNNING state, so that is
>> protecting conf. You need to do the same
>> set_config_state(DAEMON_RUNNING) in uev_trigger to make this safe.
> 
> Or not. ev_add_map calls set_config_state(DAEMON_CONFIGURE), so this
> would deadlock if you called  set_config_state(DAEMON_RUNNING) in
> uev_trigger. At any rate, either reconfigures need to be blocked while
> running uev_trigger, or we need some sort of reference counting on the
> config structure so that we don't free it when we do a reconfigure,
> until everyone has switched to the new structure.
> 
See my previous reply. We can easily move the call to
filter_devnode() so that it'll be called with the vector lock held.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH 56/57] multipathd: push down lock in checkerloop()

[Hannes Reinecke]

On 05/04/2016 12:17 AM, Benjamin Marzinski wrote:
> On Wed, Apr 27, 2016 at 01:10:57PM +0200, Hannes Reinecke wrote:
>> Instead of grabbing the lock at the start of the checkerloop
>> and releasing it at the end we should be holding it only
>> during the time when we actually need it.
> 
> I'm pretty sure that this can cause crashes if multipathd reconfigures
> when it's just started servicing a uevent. It goes like this. The
> uevq_thr thread calls uev_trigger(), which checks the running_state and
> sees that it's DAEMON_IDLE.  The uxlsnr_thr thread gets a
> reconfiguration request, which eventually calls set_config_state().
> set_config_state() sees the state is DAEMON_IDLE, sets it to
> DAEMON_CONFIGURE, and returns without waiting.  This wakes up the child
> thread, which runs reconfigure().  reconfigure() sets conf to NULL. Then
> the uevq_thr thread in uev_trigger() calls filter_devnode()
> dereferencing conf->blist_devnode and conf->elist_devnode. conf is NULL,
> and thus it crashes.
> 
> The short version is that multipathd has been using the vecs lock to
> protect conf.  With your change to uev_trigger, you access conf outside
> of the vecs lock. This isn't a problem in checkerloop(), since you can't
> reconfigure while multipathd is in the DAEMON_RUNNING state, so that is
> protecting conf. You need to do the same
> set_config_state(DAEMON_RUNNING) in uev_trigger to make this safe.
> 
I have now moved the call to 'filter_devnode' into the individual
functions, so that it will be called with the vector lock held.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[PATCH 57/57] Allow specific CLI commands to run unlocked

[Hannes Reinecke]

When multipath is busy with checking paths or processing udev
events it'll take the vector lock, causing the CLI
to become unresponsive.
This patch allows certain CLI commands to not wait for the vector
lock, so that those commands will always succeed.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 multipathd/cli.c  | 46 ++++++++++++++++++++++++++++++++++++++++++++--
 multipathd/cli.h  |  4 +++-
 multipathd/main.c | 26 ++++++++++++--------------
 3 files changed, 59 insertions(+), 17 deletions(-)

diff --git a/multipathd/cli.c b/multipathd/cli.c
index 3c9cdbf..d991cd0 100644
--- a/multipathd/cli.c
+++ b/multipathd/cli.c
@@ -1,9 +1,13 @@
 /*
  * Copyright (c) 2005 Christophe Varoqui
  */
+#include <sys/time.h>
 #include <errno.h>
+#include <pthread.h>
 #include <memory.h>
 #include <vector.h>
+#include <structs.h>
+#include <structs_vec.h>
 #include <parser.h>
 #include <util.h>
 #include <version.h>
@@ -100,6 +104,19 @@ set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *))
 	if (!h)
 		return 1;
 	h->fn = fn;
+	h->locked = 1;
+	return 0;
+}
+
+int
+set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *, char **, int *, void *))
+{
+	struct handler * h = find_handler(fp);
+
+	if (!h)
+		return 1;
+	h->fn = fn;
+	h->locked = 0;
 	return 0;
 }
 
@@ -430,11 +447,13 @@ genhelp_handler (const char *cmd, int error)
 }
 
 int
-parse_cmd (char * cmd, char ** reply, int * len, void * data)
+parse_cmd (char * cmd, char ** reply, int * len, void * data, int timeout )
 {
 	int r;
 	struct handler * h;
 	vector cmdvec = NULL;
+	struct timespec tmo;
+	struct timeval now;
 
 	r = get_cmdvec(cmd, &cmdvec);
 
@@ -456,7 +475,30 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)
 	/*
 	 * execute handler
 	 */
-	r = h->fn(cmdvec, reply, len, data);
+	if (gettimeofday(&now, NULL) == 0) {
+		tmo.tv_sec = now.tv_sec + timeout;
+		tmo.tv_nsec = now.tv_usec * 1000;
+	} else {
+		tmo.tv_sec = 0;
+	}
+	if (h->locked) {
+		struct vectors * vecs = (struct vectors *)data;
+
+		pthread_cleanup_push(cleanup_lock, &vecs->lock);
+		if (tmo.tv_sec) {
+			vecs->lock.depth++;
+			r = pthread_mutex_timedlock(vecs->lock.mutex, &tmo);
+		} else {
+			lock(vecs->lock);
+			r = 0;
+		}
+		if (r == 0) {
+			pthread_testcancel();
+			r = h->fn(cmdvec, reply, len, data);
+		}
+		lock_cleanup_pop(vecs->lock);
+	} else
+		r = h->fn(cmdvec, reply, len, data);
 	free_keys(cmdvec);
 
 	return r;
diff --git a/multipathd/cli.h b/multipathd/cli.h
index 2aa19d5..84ca40f 100644
--- a/multipathd/cli.h
+++ b/multipathd/cli.h
@@ -100,13 +100,15 @@ struct key {
 
 struct handler {
 	uint64_t fingerprint;
+	int locked;
 	int (*fn)(void *, char **, int *, void *);
 };
 
 int alloc_handlers (void);
 int add_handler (uint64_t fp, int (*fn)(void *, char **, int *, void *));
 int set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
-int parse_cmd (char * cmd, char ** reply, int * len, void *);
+int set_unlocked_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
+int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
 int load_keys (void);
 char * get_keyparam (vector v, uint64_t code);
 void free_keys (vector vec);
diff --git a/multipathd/main.c b/multipathd/main.c
index 132101f..dc788c5 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -974,14 +974,13 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data)
 	*len = 0;
 	vecs = (struct vectors *)trigger_data;
 
-	pthread_cleanup_push(cleanup_lock, &vecs->lock);
-	lock(vecs->lock);
-	pthread_testcancel();
-
-	r = parse_cmd(str, reply, len, vecs);
+	r = parse_cmd(str, reply, len, vecs, uxsock_timeout / 1000);
 
 	if (r > 0) {
-		*reply = STRDUP("fail\n");
+		if (r == ETIMEDOUT)
+			*reply = STRDUP("timeout\n");
+		else
+			*reply = STRDUP("fail\n");
 		*len = strlen(*reply) + 1;
 		r = 1;
 	}
@@ -992,7 +991,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data)
 	}
 	/* else if (r < 0) leave *reply alone */
 
-	lock_cleanup_pop(vecs->lock);
 	return r;
 }
 
@@ -1112,8 +1110,8 @@ uxlsnrloop (void * ap)
 	set_handler_callback(LIST+PATHS+RAW+FMT, cli_list_paths_raw);
 	set_handler_callback(LIST+PATH, cli_list_path);
 	set_handler_callback(LIST+MAPS, cli_list_maps);
-	set_handler_callback(LIST+STATUS, cli_list_status);
-	set_handler_callback(LIST+DAEMON, cli_list_daemon);
+	set_unlocked_handler_callback(LIST+STATUS, cli_list_status);
+	set_unlocked_handler_callback(LIST+DAEMON, cli_list_daemon);
 	set_handler_callback(LIST+MAPS+STATUS, cli_list_maps_status);
 	set_handler_callback(LIST+MAPS+STATS, cli_list_maps_stats);
 	set_handler_callback(LIST+MAPS+FMT, cli_list_maps_fmt);
@@ -1123,8 +1121,8 @@ uxlsnrloop (void * ap)
 	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
 	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
 	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
-	set_handler_callback(LIST+CONFIG, cli_list_config);
-	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
+	set_unlocked_handler_callback(LIST+CONFIG, cli_list_config);
+	set_unlocked_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
 	set_handler_callback(LIST+DEVICES, cli_list_devices);
 	set_handler_callback(LIST+WILDCARDS, cli_list_wildcards);
 	set_handler_callback(ADD+PATH, cli_add_path);
@@ -1132,7 +1130,7 @@ uxlsnrloop (void * ap)
 	set_handler_callback(ADD+MAP, cli_add_map);
 	set_handler_callback(DEL+MAP, cli_del_map);
 	set_handler_callback(SWITCH+MAP+GROUP, cli_switch_group);
-	set_handler_callback(RECONFIGURE, cli_reconfigure);
+	set_unlocked_handler_callback(RECONFIGURE, cli_reconfigure);
 	set_handler_callback(SUSPEND+MAP, cli_suspend);
 	set_handler_callback(RESUME+MAP, cli_resume);
 	set_handler_callback(RESIZE+MAP, cli_resize);
@@ -1144,8 +1142,8 @@ uxlsnrloop (void * ap)
 	set_handler_callback(RESTOREQ+MAP, cli_restore_queueing);
 	set_handler_callback(DISABLEQ+MAPS, cli_disable_all_queueing);
 	set_handler_callback(RESTOREQ+MAPS, cli_restore_all_queueing);
-	set_handler_callback(QUIT, cli_quit);
-	set_handler_callback(SHUTDOWN, cli_shutdown);
+	set_unlocked_handler_callback(QUIT, cli_quit);
+	set_unlocked_handler_callback(SHUTDOWN, cli_shutdown);
 	set_handler_callback(GETPRSTATUS+MAP, cli_getprstatus);
 	set_handler_callback(SETPRSTATUS+MAP, cli_setprstatus);
 	set_handler_callback(UNSETPRSTATUS+MAP, cli_unsetprstatus);
-- 
2.6.6

Re: [PATCH 57/57] Allow specific CLI commands to run unlocked

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:58PM +0200, Hannes Reinecke wrote:
> When multipath is busy with checking paths or processing udev
> events it'll take the vector lock, causing the CLI
> to become unresponsive.
> This patch allows certain CLI commands to not wait for the vector
> lock, so that those commands will always succeed.

In parse_cmd(), you call pthread_mutex_timedlock(), which might not lock
the mutex, but then you unconditionally unlock it later on. Also "show
config" and "show blacklist" dereference conf, and without holding the
vecs lock, there is nothing to keep a reconfigure from happening at the
same time, and setting conf to NULL.

To elaborate on my comment on the last patch: One way to fix these two
patches is to add a read/write lock that all the threads except the
child thread grab in read mode when they are active, and drop before
they sleep, and the child holds in write mode from when it deletes conf
till it finishes load_config.  We'd need to make reloading the config
happens before grabbing the vecs lock in reconfigure, to make sure that
we keep the lock ordering consistent. But then it would be safe to
dereference conf everywhere. It would also allow you to get rid of the
DAEMON_IDLE state, and cut down on the number of things the vecs lock is
protecting, which can only help.

Another option would be to add reference counts to the config structure,
and have each thread grab a reference when they are active, and drop it
before they sleep.  That way, when reconfigure happened, the old config
wouldn't be freed until all of the threads using it slept. Doing this
would involve each thread keeping track of which version of conf they
were using. That would either require passing conf as an argument to
every function that used it (which would end up being most of the
functions), or using the __thread specifier to have a thread-local conf.

-Ben


> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  multipathd/cli.c  | 46 ++++++++++++++++++++++++++++++++++++++++++++--
>  multipathd/cli.h  |  4 +++-
>  multipathd/main.c | 26 ++++++++++++--------------
>  3 files changed, 59 insertions(+), 17 deletions(-)
> 
> diff --git a/multipathd/cli.c b/multipathd/cli.c
> index 3c9cdbf..d991cd0 100644
> --- a/multipathd/cli.c
> +++ b/multipathd/cli.c
> @@ -1,9 +1,13 @@
>  /*
>   * Copyright (c) 2005 Christophe Varoqui
>   */
> +#include <sys/time.h>
>  #include <errno.h>
> +#include <pthread.h>
>  #include <memory.h>
>  #include <vector.h>
> +#include <structs.h>
> +#include <structs_vec.h>
>  #include <parser.h>
>  #include <util.h>
>  #include <version.h>
> @@ -100,6 +104,19 @@ set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *))
>  	if (!h)
>  		return 1;
>  	h->fn = fn;
> +	h->locked = 1;
> +	return 0;
> +}
> +
> +int
> +set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *, char **, int *, void *))
> +{
> +	struct handler * h = find_handler(fp);
> +
> +	if (!h)
> +		return 1;
> +	h->fn = fn;
> +	h->locked = 0;
>  	return 0;
>  }
>  
> @@ -430,11 +447,13 @@ genhelp_handler (const char *cmd, int error)
>  }
>  
>  int
> -parse_cmd (char * cmd, char ** reply, int * len, void * data)
> +parse_cmd (char * cmd, char ** reply, int * len, void * data, int timeout )
>  {
>  	int r;
>  	struct handler * h;
>  	vector cmdvec = NULL;
> +	struct timespec tmo;
> +	struct timeval now;
>  
>  	r = get_cmdvec(cmd, &cmdvec);
>  
> @@ -456,7 +475,30 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)
>  	/*
>  	 * execute handler
>  	 */
> -	r = h->fn(cmdvec, reply, len, data);
> +	if (gettimeofday(&now, NULL) == 0) {
> +		tmo.tv_sec = now.tv_sec + timeout;
> +		tmo.tv_nsec = now.tv_usec * 1000;
> +	} else {
> +		tmo.tv_sec = 0;
> +	}
> +	if (h->locked) {
> +		struct vectors * vecs = (struct vectors *)data;
> +
> +		pthread_cleanup_push(cleanup_lock, &vecs->lock);
> +		if (tmo.tv_sec) {
> +			vecs->lock.depth++;
> +			r = pthread_mutex_timedlock(vecs->lock.mutex, &tmo);
> +		} else {
> +			lock(vecs->lock);
> +			r = 0;
> +		}
> +		if (r == 0) {
> +			pthread_testcancel();
> +			r = h->fn(cmdvec, reply, len, data);
> +		}
> +		lock_cleanup_pop(vecs->lock);
> +	} else
> +		r = h->fn(cmdvec, reply, len, data);
>  	free_keys(cmdvec);
>  
>  	return r;
> diff --git a/multipathd/cli.h b/multipathd/cli.h
> index 2aa19d5..84ca40f 100644
> --- a/multipathd/cli.h
> +++ b/multipathd/cli.h
> @@ -100,13 +100,15 @@ struct key {
>  
>  struct handler {
>  	uint64_t fingerprint;
> +	int locked;
>  	int (*fn)(void *, char **, int *, void *);
>  };
>  
>  int alloc_handlers (void);
>  int add_handler (uint64_t fp, int (*fn)(void *, char **, int *, void *));
>  int set_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
> -int parse_cmd (char * cmd, char ** reply, int * len, void *);
> +int set_unlocked_handler_callback (uint64_t fp, int (*fn)(void *, char **, int *, void *));
> +int parse_cmd (char * cmd, char ** reply, int * len, void *, int);
>  int load_keys (void);
>  char * get_keyparam (vector v, uint64_t code);
>  void free_keys (vector vec);
> diff --git a/multipathd/main.c b/multipathd/main.c
> index 132101f..dc788c5 100644
> --- a/multipathd/main.c
> +++ b/multipathd/main.c
> @@ -974,14 +974,13 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data)
>  	*len = 0;
>  	vecs = (struct vectors *)trigger_data;
>  
> -	pthread_cleanup_push(cleanup_lock, &vecs->lock);
> -	lock(vecs->lock);
> -	pthread_testcancel();
> -
> -	r = parse_cmd(str, reply, len, vecs);
> +	r = parse_cmd(str, reply, len, vecs, uxsock_timeout / 1000);
>  
>  	if (r > 0) {
> -		*reply = STRDUP("fail\n");
> +		if (r == ETIMEDOUT)
> +			*reply = STRDUP("timeout\n");
> +		else
> +			*reply = STRDUP("fail\n");
>  		*len = strlen(*reply) + 1;
>  		r = 1;
>  	}
> @@ -992,7 +991,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data)
>  	}
>  	/* else if (r < 0) leave *reply alone */
>  
> -	lock_cleanup_pop(vecs->lock);
>  	return r;
>  }
>  
> @@ -1112,8 +1110,8 @@ uxlsnrloop (void * ap)
>  	set_handler_callback(LIST+PATHS+RAW+FMT, cli_list_paths_raw);
>  	set_handler_callback(LIST+PATH, cli_list_path);
>  	set_handler_callback(LIST+MAPS, cli_list_maps);
> -	set_handler_callback(LIST+STATUS, cli_list_status);
> -	set_handler_callback(LIST+DAEMON, cli_list_daemon);
> +	set_unlocked_handler_callback(LIST+STATUS, cli_list_status);
> +	set_unlocked_handler_callback(LIST+DAEMON, cli_list_daemon);
>  	set_handler_callback(LIST+MAPS+STATUS, cli_list_maps_status);
>  	set_handler_callback(LIST+MAPS+STATS, cli_list_maps_stats);
>  	set_handler_callback(LIST+MAPS+FMT, cli_list_maps_fmt);
> @@ -1123,8 +1121,8 @@ uxlsnrloop (void * ap)
>  	set_handler_callback(LIST+MAP+TOPOLOGY, cli_list_map_topology);
>  	set_handler_callback(LIST+MAP+FMT, cli_list_map_fmt);
>  	set_handler_callback(LIST+MAP+RAW+FMT, cli_list_map_fmt);
> -	set_handler_callback(LIST+CONFIG, cli_list_config);
> -	set_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
> +	set_unlocked_handler_callback(LIST+CONFIG, cli_list_config);
> +	set_unlocked_handler_callback(LIST+BLACKLIST, cli_list_blacklist);
>  	set_handler_callback(LIST+DEVICES, cli_list_devices);
>  	set_handler_callback(LIST+WILDCARDS, cli_list_wildcards);
>  	set_handler_callback(ADD+PATH, cli_add_path);
> @@ -1132,7 +1130,7 @@ uxlsnrloop (void * ap)
>  	set_handler_callback(ADD+MAP, cli_add_map);
>  	set_handler_callback(DEL+MAP, cli_del_map);
>  	set_handler_callback(SWITCH+MAP+GROUP, cli_switch_group);
> -	set_handler_callback(RECONFIGURE, cli_reconfigure);
> +	set_unlocked_handler_callback(RECONFIGURE, cli_reconfigure);
>  	set_handler_callback(SUSPEND+MAP, cli_suspend);
>  	set_handler_callback(RESUME+MAP, cli_resume);
>  	set_handler_callback(RESIZE+MAP, cli_resize);
> @@ -1144,8 +1142,8 @@ uxlsnrloop (void * ap)
>  	set_handler_callback(RESTOREQ+MAP, cli_restore_queueing);
>  	set_handler_callback(DISABLEQ+MAPS, cli_disable_all_queueing);
>  	set_handler_callback(RESTOREQ+MAPS, cli_restore_all_queueing);
> -	set_handler_callback(QUIT, cli_quit);
> -	set_handler_callback(SHUTDOWN, cli_shutdown);
> +	set_unlocked_handler_callback(QUIT, cli_quit);
> +	set_unlocked_handler_callback(SHUTDOWN, cli_shutdown);
>  	set_handler_callback(GETPRSTATUS+MAP, cli_getprstatus);
>  	set_handler_callback(SETPRSTATUS+MAP, cli_setprstatus);
>  	set_handler_callback(UNSETPRSTATUS+MAP, cli_unsetprstatus);
> -- 
> 2.6.6

Re: [PATCH 00/57] SLES resync

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 10979 bytes --]

Here is the patchset merged so far.

e0c1b55e171373d9efbde43502a9c87fdb53f0ef Allow specific CLI commands to run
unlocked
5ed6122c0e5d8326bc0f3beaf1f5de7303986952 multipathd: push down lock in
checkerloop()
5aca183c68f7ada6fd38c866e235abc690e24328 multipathd: asynchronous
configuration
996b564931e92b5864a3e740b6b5198d69be17da multipathd: Increase uxclnt timeout
c50c729f6c72df0c164e42bde1cc94114c6c06ea multipathd: implement 'show map
$map format $fmt'
c263e76be2f83bfb94f8aecd47a3e345c2d79fc6 libmultipath: make 'dm_addmap'
static
2c9c69956431f07942c4ca71d9f1d779982d62ac multipathd: Provide standard error
description on cli failure
5007e197c55d8a287ac5ec84c1f13aef7a7a08db multipathd: strict loop timings
c0db476f2b0d92b3ab4ea5007242122fdbb103dc dmparser: Use find_path_by_dev()
18f1b0341f321f9326e08b940b3c1eed4a31d779 libmultipath: sanity check on
store_path()
3ede290fd5267176860212483f9eaad48a5e5ca8 libmultipath: ensure 'dev_t' is
set when store paths
1be502f61f92b93ab8eb1c3d84e2c19423408787 libmultipath: remove 'get_info'
argument for adopt_paths()
d87f3055b465743d00d6cae3f4d53c4cc171f2ab multipathd: Do not switch paths on
empty multipath tables
c6db07310a4c63b65a758892de187837bb381572 multipathd: set uxsock_timeout
after reconfiguration
a5b3c5561fd5300046d72f7ce4b414e8acce8547 Use multipath wwid if path wwid is
empty
0c833dd35e98da8e6a58c8a50ae6a08cd0e07e20 multipathd: accept zero-size paths
in ev_add_path()
26c4bb0f89ba9bb15e6f1873799a762e6e5078a7 Always set
DM_UDEV_DISABLE_LIBRARY_FALLBACK
a632ef41d98b4369e105ab630ffa2679a98014d3 libmultipath: remove 'use_uuid'
argument from dm_addmap()
74cbb841cec6273243d3949b277adb2430a9d6aa libmultipath: remove 'needsync'
argument from dm_simplecmd_noflush
6e14b7b54de497405878ad92c60813bf14cd4fd8 multipathd: improve uxlsnr
42539b051468bd442f5f0365b178cd9530da12f3 multipathd: skip uninitialized
devices during reconfiguration
10bd53368193cc009bdf6e52f9f01fb9ced98045 Only filter for udev property if
uid_attribute is present
5ec07b3af8d1c3a6c7ba1680af20b80ddce4f962 libmultipath: use a shared lock to
co-operate with udev
cc37c9941a2e343b0bf58c8dfef28dab3634be6f Ensure multipathd is started
before systemd-udev-trigger.service
286749f8ba07e198ad314f0d6a4d1afe4ecd7332 11-dm-mpath.rules: Only import
ID_FS_XXX variables if not set
85398f6c8f91e8e4d1a2d0fad0d603aa5521ec26 Load all device handler modules on
startup
2ccc1a70f294e9d143b78864a943e419d93a7ea0 Add dependency on
systemd-udevd.service
3f05ebe105464ed520397d5dd8b5448338111f5e libmultipath: avoid double
semicolon in lock.h
505227acb951dc0ad063d2322036cd894b9556ab multipathd: Do not print
misleading message 'not found in pathvec'
4cdbd2990f42431b9c19866e62d2ddd723697487 libmpathpersist: Fixup whitespaces
in Makefile
cb7e5658c2626b9db70528255d419d8a6ba89601 libmultipath: additional logging
messages when formatting callout
79f88dd35ab1be62f7e7d33c281a477e5346c2c3 libmultipath: call get_uid() for
all paths
6903495e748d938d8c109bd3b9e4ad6ba4004049 libmultipath: correctly display
checker status
b4d9ca8dc8bbfbd3782bf4cf2cb1a440685ccd07 libmultipath: correctly initialize
pp->sg_id
d291112dd24fe4eff70dfd50ca817146fc100936 multipathd: print error message
for invalid arguments
42f490c62b066235693a1176204fa1712456fd4d multipathd: fixup a crash when
invoking CLI commands
2516c4f1dd5c85799fff3d5c7cb5de54dece88ba multipathd: fixup queueing mode in
'show maps status'
ccf6e5316a7cd090bfa2ba4c542859f55e20c8d9 libmultipath: finally fix
dev_loss_tmo setting
8af1770fcdd9ebbd6bcb29baa49f6d0de1b1b700 libmultipath: Do not use 'sscanf'
for parsing integers
b4e1b5242355908fcb1bd461efb2851a0b41dba7 Allow for empty SCSI revision
6fe55f63b5a95eb9d4da5e32115e1629784619af Add LIO-ORG/SUSE RBD backend
hardware defaults
21136f36a38ce761a76797a2434c4fbbcbc89830 Add HP MSA 2040 to the hardware
table
c416b69808e5bc80bfb4788e3d5309c6d5b4b5b5 kpartx: Install rules file with
correct prefix
adfc5ad9713ba58d44e2d50ed8725f6a72127239 kpartx: parse emulated DASD devices
0d83956858a946817cd09b818d70de010f7bdd76 kpartx: handle more than 256 loop
devices
08cc0ccd943b35c0aaedac44632dc8056f451f41 kpartx: Fixup persistent name
generation

4628ca72bf953a26bc3580daa61e06ec1fbf4fa1 multipath: remove warning 'failed
to get wwid'
is replaced with:
01b0f396797f008b63fc244aee2b7c79e9a0f41e multipath: move the warning
'failed to get wwid' to lvl4 verbosity

Thanks.
Christophe

On Wed, Apr 27, 2016 at 1:10 PM, Hannes Reinecke <hare@suse.de> wrote:

> Hi all,
>
> as promised several times, here's now a patchset with all the patches
> queued up in my SLES repository.
>
> The first bits are pretty much uncontroversial (I hope), mainly fixes
> etc. Next there are some systemd updates, setting dependencies and such.
>
> The _really_ important patch is
>
> multipathd: skip uninitialized devices during reconfiguration
>
> which basically is the cause for all the missing wwid issues we've seen.
> So I wonder if we shouldn't just do away with all the tweaks we've
> accumlated so far and see if that patch isn't sufficient.
> According to Kay Sievers it should...
>
> Next is a general overhaul of the 'cookie' mechanism; Ben, Mike,
> mind to have a look here? It _seems_ reasonable enough, but it would
> be nice to have additional confirmation for that.
>
> Finally there are patches which probably require some more discussion:
> - Asynchronous configuration: Do not call 'reconfigure' directly, but
>   rather set the daemon_status to 'configuration' and defer the actual
>   work to the main thread. This allows us to decouple it from the CLI
>   call, and we won't run into any udev/systemd timeout issues when
>   booting on large installations
> - Lock pushdown: I remember to have sent this patch already at one
>   point, but it got rejected. So I try again :-)
>   With this patch the big vector lock is pushed down to the areas
>   where we actually need it, making the entire system running far
>   smoother during booting or failover in large instances.
>   We have been using this patch since SLE12 GA, ie for over two
>   years now. And seem to have ironed out all issues with it.
>
> The entire patchset can be found at:
>
> github.com:/hreinecke/multipath-tools.git
> branch sles-resync
>
> As usual, comments and reviews are welcome.
> Patches are based upon the latest git, including Bens patchset.
>
> Hannes Reinecke (57):
>   kpartx: Fixup persistent name generation
>   kpartx: handle more than 256 loop devices
>   kpartx: parse emulated DASD devices
>   kpartx: Install rules file with correct prefix
>   Add HP MSA 2040 to the hardware table
>   Use ALUA for HP 3PAR
>   Add LIO-ORG/SUSE RBD backend hardware defaults
>   Allow for empty SCSI revision
>   libmultipath: Do not use 'sscanf' for parsing integers
>   libmultipath: finally fix dev_loss_tmo setting
>   multipathd: fixup queueing mode in 'show maps status'
>   multipathd: fixup a crash when invoking CLI commands
>   multipathd: print error message for invalid arguments
>   libmultipath: correctly initialize pp->sg_id
>   libmultipath: correctly display checker status
>   libmultipath: call get_uid() for all paths
>   libmultipath: additional logging messages when formatting callout
>   libmpathpersist: Fixup whitespaces in Makefile
>   multipathd: Do not print misleading message 'not found in pathvec'
>   multipathd: Do not update the paths vec when removing paths
>   libmultipath: avoid double semicolon in lock.h
>   multipath: use option '-i' when called from udev
>   multipath: remove warning 'failed to get wwid'
>   Add dependency on systemd-udevd.service
>   Load all device handler modules on startup
>   11-dm-mpath.rules: Only import ID_FS_XXX variables if not set
>   Ensure multipathd is started before systemd-udev-trigger.service
>   libmultipath: use a shared lock to co-operate with udev
>   Only filter for udev property if uid_attribute is present
>   multipathd: skip uninitialized devices during reconfiguration
>   multipathd: improve uxlsnr
>   libmultipath: remove 'needsync' argument from dm_simplecmd_noflush
>   libmultipath: remove 'use_uuid' argument from dm_addmap()
>   Always set DM_UDEV_DISABLE_LIBRARY_FALLBACK
>   libmultipath: pass in cookie as argument for dm_simplecmd()
>   libmultipath: pass in 'cookie' as argument for dm_addmap()
>   Remove 'udev_sync' argument from dm_simplecmd()
>   libmultipath: Fixup 'DM_DEVICE_RELOAD' handling
>   devmapper: do not flush I/O for DM_DEVICE_CREATE
>   libmultipath: fixup dm_rename to complete cookie on failure
>   multipathd: accept zero-size paths in ev_add_path()
>   Use multipath wwid if path wwid is empty
>   multipathd: set uxsock_timeout after reconfiguration
>   multipathd: Do not switch paths on empty multipath tables
>   libmultipath: remove 'get_info' argument for adopt_paths()
>   libmultipath: ensure 'dev_t' is set when store paths
>   libmultipath: sanity check on store_path()
>   dmparser: Use find_path_by_dev()
>   multipathd: do not flush maps on startup
>   multipathd: strict loop timings
>   multipathd: Provide standard error description on cli failure
>   libmultipath: make 'dm_addmap' static
>   multipathd: implement 'show map $map format $fmt'
>   multipathd: Increase uxclnt timeout
>   multipathd: asynchronous configuration
>   multipathd: push down lock in checkerloop()
>   Allow specific CLI commands to run unlocked
>
>  kpartx/Makefile               |   6 +-
>  kpartx/dasd.c                 |  39 +++-
>  kpartx/dasd.h                 |   1 +
>  kpartx/kpartx.rules           |  11 +-
>  kpartx/kpartx_id              |  11 +-
>  kpartx/lopart.c               | 106 +++++----
>  libmpathpersist/Makefile      |   4 +-
>  libmultipath/callout.c        |   6 +-
>  libmultipath/config.h         |   1 +
>  libmultipath/configure.c      |  33 ++-
>  libmultipath/defaults.h       |   1 +
>  libmultipath/devmapper.c      | 114 ++++++----
>  libmultipath/devmapper.h      |   4 +-
>  libmultipath/dict.c           |   4 +
>  libmultipath/discovery.c      | 115 +++++-----
>  libmultipath/dmparser.c       |  20 +-
>  libmultipath/hwtable.c        |  41 +++-
>  libmultipath/lock.h           |   4 +-
>  libmultipath/print.c          |   6 +-
>  libmultipath/structs.c        |  17 +-
>  libmultipath/structs_vec.c    |  10 +-
>  libmultipath/structs_vec.h    |   2 +-
>  libmultipath/uevent.c         |  10 +-
>  multipath/11-dm-mpath.rules   |  10 +-
>  multipath/main.c              |   1 -
>  multipath/multipath.conf.5    |  16 +-
>  multipath/multipath.rules     |   2 +-
>  multipathd/cli.c              | 101 +++++++--
>  multipathd/cli.h              |   4 +-
>  multipathd/cli_handlers.c     |  85 +++++++-
>  multipathd/cli_handlers.h     |   2 +
>  multipathd/main.c             | 486
> +++++++++++++++++++++++++++++-------------
>  multipathd/main.h             |   3 +
>  multipathd/multipathd.service |   6 +-
>  multipathd/uxlsnr.c           |  97 +++++----
>  35 files changed, 923 insertions(+), 456 deletions(-)
>
> --
> 2.6.6
>
>

[-- Attachment #1.2: Type: text/html, Size: 12912 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH 00/57] SLES resync

[Benjamin Marzinski]

On Wed, Apr 27, 2016 at 01:10:01PM +0200, Hannes Reinecke wrote:
> Hi all,
> 
> as promised several times, here's now a patchset with all the patches
> queued up in my SLES repository.

ACK on all the patches I haven't commented on.

-Ben

> 
> The first bits are pretty much uncontroversial (I hope), mainly fixes
> etc. Next there are some systemd updates, setting dependencies and such.
> 
> The _really_ important patch is
> 
> multipathd: skip uninitialized devices during reconfiguration
> 
> which basically is the cause for all the missing wwid issues we've seen.
> So I wonder if we shouldn't just do away with all the tweaks we've
> accumlated so far and see if that patch isn't sufficient.
> According to Kay Sievers it should...
> 
> Next is a general overhaul of the 'cookie' mechanism; Ben, Mike,
> mind to have a look here? It _seems_ reasonable enough, but it would
> be nice to have additional confirmation for that.
> 
> Finally there are patches which probably require some more discussion:
> - Asynchronous configuration: Do not call 'reconfigure' directly, but
>   rather set the daemon_status to 'configuration' and defer the actual
>   work to the main thread. This allows us to decouple it from the CLI
>   call, and we won't run into any udev/systemd timeout issues when
>   booting on large installations
> - Lock pushdown: I remember to have sent this patch already at one
>   point, but it got rejected. So I try again :-)
>   With this patch the big vector lock is pushed down to the areas
>   where we actually need it, making the entire system running far
>   smoother during booting or failover in large instances.
>   We have been using this patch since SLE12 GA, ie for over two
>   years now. And seem to have ironed out all issues with it.
> 
> The entire patchset can be found at:
> 
> github.com:/hreinecke/multipath-tools.git
> branch sles-resync
> 
> As usual, comments and reviews are welcome.
> Patches are based upon the latest git, including Bens patchset.
> 
> Hannes Reinecke (57):
>   kpartx: Fixup persistent name generation
>   kpartx: handle more than 256 loop devices
>   kpartx: parse emulated DASD devices
>   kpartx: Install rules file with correct prefix
>   Add HP MSA 2040 to the hardware table
>   Use ALUA for HP 3PAR
>   Add LIO-ORG/SUSE RBD backend hardware defaults
>   Allow for empty SCSI revision
>   libmultipath: Do not use 'sscanf' for parsing integers
>   libmultipath: finally fix dev_loss_tmo setting
>   multipathd: fixup queueing mode in 'show maps status'
>   multipathd: fixup a crash when invoking CLI commands
>   multipathd: print error message for invalid arguments
>   libmultipath: correctly initialize pp->sg_id
>   libmultipath: correctly display checker status
>   libmultipath: call get_uid() for all paths
>   libmultipath: additional logging messages when formatting callout
>   libmpathpersist: Fixup whitespaces in Makefile
>   multipathd: Do not print misleading message 'not found in pathvec'
>   multipathd: Do not update the paths vec when removing paths
>   libmultipath: avoid double semicolon in lock.h
>   multipath: use option '-i' when called from udev
>   multipath: remove warning 'failed to get wwid'
>   Add dependency on systemd-udevd.service
>   Load all device handler modules on startup
>   11-dm-mpath.rules: Only import ID_FS_XXX variables if not set
>   Ensure multipathd is started before systemd-udev-trigger.service
>   libmultipath: use a shared lock to co-operate with udev
>   Only filter for udev property if uid_attribute is present
>   multipathd: skip uninitialized devices during reconfiguration
>   multipathd: improve uxlsnr
>   libmultipath: remove 'needsync' argument from dm_simplecmd_noflush
>   libmultipath: remove 'use_uuid' argument from dm_addmap()
>   Always set DM_UDEV_DISABLE_LIBRARY_FALLBACK
>   libmultipath: pass in cookie as argument for dm_simplecmd()
>   libmultipath: pass in 'cookie' as argument for dm_addmap()
>   Remove 'udev_sync' argument from dm_simplecmd()
>   libmultipath: Fixup 'DM_DEVICE_RELOAD' handling
>   devmapper: do not flush I/O for DM_DEVICE_CREATE
>   libmultipath: fixup dm_rename to complete cookie on failure
>   multipathd: accept zero-size paths in ev_add_path()
>   Use multipath wwid if path wwid is empty
>   multipathd: set uxsock_timeout after reconfiguration
>   multipathd: Do not switch paths on empty multipath tables
>   libmultipath: remove 'get_info' argument for adopt_paths()
>   libmultipath: ensure 'dev_t' is set when store paths
>   libmultipath: sanity check on store_path()
>   dmparser: Use find_path_by_dev()
>   multipathd: do not flush maps on startup
>   multipathd: strict loop timings
>   multipathd: Provide standard error description on cli failure
>   libmultipath: make 'dm_addmap' static
>   multipathd: implement 'show map $map format $fmt'
>   multipathd: Increase uxclnt timeout
>   multipathd: asynchronous configuration
>   multipathd: push down lock in checkerloop()
>   Allow specific CLI commands to run unlocked
> 
>  kpartx/Makefile               |   6 +-
>  kpartx/dasd.c                 |  39 +++-
>  kpartx/dasd.h                 |   1 +
>  kpartx/kpartx.rules           |  11 +-
>  kpartx/kpartx_id              |  11 +-
>  kpartx/lopart.c               | 106 +++++----
>  libmpathpersist/Makefile      |   4 +-
>  libmultipath/callout.c        |   6 +-
>  libmultipath/config.h         |   1 +
>  libmultipath/configure.c      |  33 ++-
>  libmultipath/defaults.h       |   1 +
>  libmultipath/devmapper.c      | 114 ++++++----
>  libmultipath/devmapper.h      |   4 +-
>  libmultipath/dict.c           |   4 +
>  libmultipath/discovery.c      | 115 +++++-----
>  libmultipath/dmparser.c       |  20 +-
>  libmultipath/hwtable.c        |  41 +++-
>  libmultipath/lock.h           |   4 +-
>  libmultipath/print.c          |   6 +-
>  libmultipath/structs.c        |  17 +-
>  libmultipath/structs_vec.c    |  10 +-
>  libmultipath/structs_vec.h    |   2 +-
>  libmultipath/uevent.c         |  10 +-
>  multipath/11-dm-mpath.rules   |  10 +-
>  multipath/main.c              |   1 -
>  multipath/multipath.conf.5    |  16 +-
>  multipath/multipath.rules     |   2 +-
>  multipathd/cli.c              | 101 +++++++--
>  multipathd/cli.h              |   4 +-
>  multipathd/cli_handlers.c     |  85 +++++++-
>  multipathd/cli_handlers.h     |   2 +
>  multipathd/main.c             | 486 +++++++++++++++++++++++++++++-------------
>  multipathd/main.h             |   3 +
>  multipathd/multipathd.service |   6 +-
>  multipathd/uxlsnr.c           |  97 +++++----
>  35 files changed, 923 insertions(+), 456 deletions(-)
> 
> -- 
> 2.6.6