[PATCHv4 00/23] asynchronous ALUA device handler

[PATCHv4 00/23] asynchronous ALUA device handler

[Hannes Reinecke]

Hi all,

here is an update to the ALUA device handler. The main
features are:

- Topology discovery: the device handler creates a separate
  port_group structure, which is used to update all paths to
  the same port group. With that we achieve a significant
  reduction of the number of RTPGs.
- Asynchronous state update: The ALUA state is now updated
  from a workqueue item, so all concurrent RTPG calls are
  coaleasced. The ALUA state update is also triggered by
  sense codes indicating an ALUA state change.
- Use the existing vpd page 0x83 to detect device IDs

The patchset is relative to the V4 version of the scsi_dh update.
The entire tree can be found at

kernel/hare/scsi-devel branch alua.v4

on git.kernel.org

As usual, reviews and comments are welcome.

Changes to v1:
  - Split off rtpg workqueue into separate items
  - User per-port workqueues
  - Incorporate review from Bart
  - Incorporate review from hch
Changes to v2:
  - Fixup patch descriptions
  - Split off 'Improve error handling' patch
Changes to v3:
  - Implement generic 'scsi_vpd_lun_id()' function to parse vpd page 0x83

Hannes Reinecke (23):
  scsi_dh_alua: Disable ALUA handling for non-disk devices
  scsi_dh_alua: Use vpd_pg83 information
  scsi_dh_alua: improved logging
  scsi_dh_alua: use standard logging functions
  scsi_dh_alua: return standard SCSI return codes in submit_rtpg
  scsi_dh_alua: fixup description of stpg_endio()
  scsi: remove scsi_show_sense_hdr()
  scsi_dh_alua: use flag for RTPG extended header
  scsi_dh_alua: use unaligned access macros
  scsi_dh_alua: Pass buffer as function argument
  scsi_dh_alua: Make stpg synchronous
  scsi_dh_alua: switch to scsi_execute_req_flags()
  scsi_dh_alua: Use separate alua_port_group structure
  scsi_dh_alua: allocate RTPG buffer separately
  scsi_dh_alua: simplify sense code handling
  scsi: Add scsi_vpd_lun_id()
  scsi_dh_alua: use unique device id
  revert "scsi_dh_alua: ALUA hander attach should succeed while TPG is
    transitioning"
  scsi_dh_alua: Use workqueue for RTPG
  scsi_dh_alua: Recheck state on unit attention
  scsi_dh_alua: update all port states
  scsi_dh_alua: Send TEST UNIT READY to poll for transitioning
  scsi_dh_alua: Update version to 2.0

 drivers/scsi/device_handler/scsi_dh_alua.c | 1140 +++++++++++++++++-----------
 drivers/scsi/scsi_lib.c                    |  121 +++
 include/scsi/scsi_dbg.h                    |    2 -
 include/scsi/scsi_device.h                 |    1 +
 4 files changed, 838 insertions(+), 426 deletions(-)

-- 
1.8.5.6

[PATCH 01/23] scsi_dh_alua: Disable ALUA handling for non-disk devices

[Hannes Reinecke]

Non-disk devices might support ALUA, but the firmware
implementation is untested and frequently broken.
As we're don't actually need it disable ALUA support
for non-disk device for now.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index cc2773b..7d01ef0 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -320,6 +320,18 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
 {
 	int err = SCSI_DH_OK;
 
+	/*
+	 * ALUA support for non-disk devices is fraught with
+	 * difficulties, so disable it for now.
+	 */
+	if (sdev->type != TYPE_DISK) {
+		h->tpgs = TPGS_MODE_NONE;
+		sdev_printk(KERN_INFO, sdev,
+			    "%s: disable for non-disk devices\n",
+			    ALUA_DH_NAME);
+		return SCSI_DH_DEV_UNSUPP;
+	}
+
 	h->tpgs = scsi_device_tpgs(sdev);
 	switch (h->tpgs) {
 	case TPGS_MODE_EXPLICIT|TPGS_MODE_IMPLICIT:
-- 
1.8.5.6

[PATCH 02/23] scsi_dh_alua: Use vpd_pg83 information

[Hannes Reinecke]

The SCSI device now has the VPD page 0x83 information attached,
so there is no need to query it again.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 83 +++++-------------------------
 1 file changed, 13 insertions(+), 70 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 7d01ef0..f15b977 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -131,43 +131,6 @@ static struct request *get_alua_req(struct scsi_device *sdev,
 }
 
 /*
- * submit_vpd_inquiry - Issue an INQUIRY VPD page 0x83 command
- * @sdev: sdev the command should be sent to
- */
-static int submit_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
-{
-	struct request *rq;
-	int err = SCSI_DH_RES_TEMP_UNAVAIL;
-
-	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
-	if (!rq)
-		goto done;
-
-	/* Prepare the command. */
-	rq->cmd[0] = INQUIRY;
-	rq->cmd[1] = 1;
-	rq->cmd[2] = 0x83;
-	rq->cmd[4] = h->bufflen;
-	rq->cmd_len = COMMAND_SIZE(INQUIRY);
-
-	rq->sense = h->sense;
-	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
-	rq->sense_len = h->senselen = 0;
-
-	err = blk_execute_rq(rq->q, NULL, rq, 1);
-	if (err == -EIO) {
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: evpd inquiry failed with %x\n",
-			    ALUA_DH_NAME, rq->errors);
-		h->senselen = rq->sense_len;
-		err = SCSI_DH_IO;
-	}
-	blk_put_request(rq);
-done:
-	return err;
-}
-
-/*
  * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
  * @sdev: sdev the command should be sent to
  */
@@ -359,43 +322,24 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
 }
 
 /*
- * alua_vpd_inquiry - Evaluate INQUIRY vpd page 0x83
+ * alua_check_vpd - Evaluate INQUIRY vpd page 0x83
  * @sdev: device to be checked
  *
  * Extract the relative target port and the target port group
  * descriptor from the list of identificators.
  */
-static int alua_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
+static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 {
-	int len;
-	unsigned err;
 	unsigned char *d;
 
- retry:
-	err = submit_vpd_inquiry(sdev, h);
-
-	if (err != SCSI_DH_OK)
-		return err;
-
-	/* Check if vpd page exceeds initial buffer */
-	len = (h->buff[2] << 8) + h->buff[3] + 4;
-	if (len > h->bufflen) {
-		/* Resubmit with the correct length */
-		if (realloc_buffer(h, len)) {
-			sdev_printk(KERN_WARNING, sdev,
-				    "%s: kmalloc buffer failed\n",
-				    ALUA_DH_NAME);
-			/* Temporary failure, bypass */
-			return SCSI_DH_DEV_TEMP_BUSY;
-		}
-		goto retry;
-	}
+	if (!sdev->vpd_pg83)
+		return SCSI_DH_DEV_UNSUPP;
 
 	/*
-	 * Now look for the correct descriptor.
+	 * Look for the correct descriptor.
 	 */
-	d = h->buff + 4;
-	while (d < h->buff + len) {
+	d = sdev->vpd_pg83 + 4;
+	while (d < sdev->vpd_pg83 + sdev->vpd_pg83_len) {
 		switch (d[1] & 0xf) {
 		case 0x4:
 			/* Relative target port */
@@ -422,14 +366,13 @@ static int alua_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
 			    ALUA_DH_NAME);
 		h->state = TPGS_STATE_OPTIMIZED;
 		h->tpgs = TPGS_MODE_NONE;
-		err = SCSI_DH_DEV_UNSUPP;
-	} else {
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: port group %02x rel port %02x\n",
-			    ALUA_DH_NAME, h->group_id, h->rel_port);
+		return SCSI_DH_DEV_UNSUPP;
 	}
+	sdev_printk(KERN_INFO, sdev,
+		    "%s: port group %02x rel port %02x\n",
+		    ALUA_DH_NAME, h->group_id, h->rel_port);
 
-	return err;
+	return 0;
 }
 
 static char print_alua_state(int state)
@@ -692,7 +635,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
 	if (err != SCSI_DH_OK)
 		goto out;
 
-	err = alua_vpd_inquiry(sdev, h);
+	err = alua_check_vpd(sdev, h);
 	if (err != SCSI_DH_OK)
 		goto out;
 
-- 
1.8.5.6

[PATCH 03/23] scsi_dh_alua: improved logging

[Hannes Reinecke]

Issue different logging messages if ALUA is not supported
or the TPGS setting is invalid.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index f15b977..a20c8bf 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -310,12 +310,18 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
 		sdev_printk(KERN_INFO, sdev, "%s: supports implicit TPGS\n",
 			    ALUA_DH_NAME);
 		break;
-	default:
-		h->tpgs = TPGS_MODE_NONE;
+	case TPGS_MODE_NONE:
 		sdev_printk(KERN_INFO, sdev, "%s: not supported\n",
 			    ALUA_DH_NAME);
 		err = SCSI_DH_DEV_UNSUPP;
 		break;
+	default:
+		sdev_printk(KERN_INFO, sdev,
+			    "%s: unsupported TPGS setting %d\n",
+			    ALUA_DH_NAME, h->tpgs);
+		h->tpgs = TPGS_MODE_NONE;
+		err = SCSI_DH_DEV_UNSUPP;
+		break;
 	}
 
 	return err;
-- 
1.8.5.6

[PATCH 04/23] scsi_dh_alua: use standard logging functions

[Hannes Reinecke]

Use standard logging functions instead of hand-crafted ones.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 34 +++++++++++++-----------------
 1 file changed, 15 insertions(+), 19 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index a20c8bf..7b43ee3 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -23,6 +23,7 @@
 #include <linux/delay.h>
 #include <linux/module.h>
 #include <scsi/scsi.h>
+#include <scsi/scsi_dbg.h>
 #include <scsi/scsi_eh.h>
 #include <scsi/scsi_dh.h>
 
@@ -194,22 +195,16 @@ static void stpg_endio(struct request *req, int error)
 		goto done;
 	}
 
-	if (req->sense_len > 0) {
-		err = scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-					   &sense_hdr);
-		if (!err) {
-			err = SCSI_DH_IO;
-			goto done;
-		}
+	if (scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
+				 &sense_hdr)) {
 		err = alua_check_sense(h->sdev, &sense_hdr);
 		if (err == ADD_TO_MLQUEUE) {
 			err = SCSI_DH_RETRY;
 			goto done;
 		}
-		sdev_printk(KERN_INFO, h->sdev,
-			    "%s: stpg sense code: %02x/%02x/%02x\n",
-			    ALUA_DH_NAME, sense_hdr.sense_key,
-			    sense_hdr.asc, sense_hdr.ascq);
+		sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
+			    ALUA_DH_NAME);
+		scsi_print_sense_hdr(h->sdev, ALUA_DH_NAME, &sense_hdr);
 		err = SCSI_DH_IO;
 	} else if (error)
 		err = SCSI_DH_IO;
@@ -530,16 +525,17 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		}
 
 		err = alua_check_sense(sdev, &sense_hdr);
-		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry))
+		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
+			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
+				    ALUA_DH_NAME);
+			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
 			goto retry;
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: rtpg sense code %02x/%02x/%02x\n",
-			    ALUA_DH_NAME, sense_hdr.sense_key,
-			    sense_hdr.asc, sense_hdr.ascq);
-		err = SCSI_DH_IO;
+		}
+		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
+			    ALUA_DH_NAME);
+		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
+		return SCSI_DH_IO;
 	}
-	if (err != SCSI_DH_OK)
-		return err;
 
 	len = (h->buff[0] << 24) + (h->buff[1] << 16) +
 		(h->buff[2] << 8) + h->buff[3] + 4;
-- 
1.8.5.6

[PATCH 05/23] scsi_dh_alua: return standard SCSI return codes in submit_rtpg

[Hannes Reinecke]

Fixup submit_rtpg() to always return a standard SCSI return code.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 38 +++++++++++++++++-------------
 1 file changed, 22 insertions(+), 16 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 7b43ee3..c41d662 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -139,11 +139,13 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
 			    bool rtpg_ext_hdr_req)
 {
 	struct request *rq;
-	int err = SCSI_DH_RES_TEMP_UNAVAIL;
+	int err = 0;
 
 	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
-	if (!rq)
+	if (!rq) {
+		err = DRIVER_BUSY << 24;
 		goto done;
+	}
 
 	/* Prepare the command. */
 	rq->cmd[0] = MAINTENANCE_IN;
@@ -161,13 +163,10 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
 	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
 	rq->sense_len = h->senselen = 0;
 
-	err = blk_execute_rq(rq->q, NULL, rq, 1);
-	if (err == -EIO) {
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: rtpg failed with %x\n",
-			    ALUA_DH_NAME, rq->errors);
+	blk_execute_rq(rq->q, NULL, rq, 1);
+	if (rq->errors) {
+		err = rq->errors;
 		h->senselen = rq->sense_len;
-		err = SCSI_DH_IO;
 	}
 	blk_put_request(rq);
 done:
@@ -489,7 +488,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 	struct scsi_sense_hdr sense_hdr;
 	int len, k, off, valid_states = 0;
 	unsigned char *ucp;
-	unsigned err;
+	unsigned err, retval;
 	bool rtpg_ext_hdr_req = 1;
 	unsigned long expiry, interval = 0;
 	unsigned int tpg_desc_tbl_off;
@@ -501,13 +500,20 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
 
  retry:
-	err = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
-
-	if (err == SCSI_DH_IO && h->senselen > 0) {
-		err = scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-					   &sense_hdr);
-		if (!err)
-			return SCSI_DH_IO;
+	retval = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
+
+	if (retval) {
+		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
+					  &sense_hdr)) {
+			sdev_printk(KERN_INFO, sdev,
+				    "%s: rtpg failed, result %d\n",
+				    ALUA_DH_NAME, retval);
+			if (driver_byte(retval) == DRIVER_BUSY)
+				err = SCSI_DH_DEV_TEMP_BUSY;
+			else
+				err = SCSI_DH_IO;
+			return err;
+		}
 
 		/*
 		 * submit_rtpg() has failed on existing arrays
-- 
1.8.5.6

[PATCH 06/23] scsi_dh_alua: fixup description of stpg_endio()

[Hannes Reinecke]

Fixup copy-and-paste error in the description of stpg_endio().

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index c41d662..25c2045 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -174,13 +174,11 @@ done:
 }
 
 /*
- * alua_stpg - Evaluate SET TARGET GROUP STATES
+ * stpg_endio - Evaluate SET TARGET GROUP STATES
  * @sdev: the device to be evaluated
  * @state: the new target group state
  *
- * Send a SET TARGET GROUP STATES command to the device.
- * We only have to test here if we should resubmit the command;
- * any other error is assumed as a failure.
+ * Evaluate a SET TARGET GROUP STATES command response.
  */
 static void stpg_endio(struct request *req, int error)
 {
-- 
1.8.5.6

[PATCH 07/23] scsi: remove scsi_show_sense_hdr()

[Hannes Reinecke]

Last caller is gone, so remove it.

Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
Reviewed-by: Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 include/scsi/scsi_dbg.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/include/scsi/scsi_dbg.h b/include/scsi/scsi_dbg.h
index f8170e9..56710e0 100644
--- a/include/scsi/scsi_dbg.h
+++ b/include/scsi/scsi_dbg.h
@@ -12,8 +12,6 @@ extern size_t __scsi_format_command(char *, size_t,
 				   const unsigned char *, size_t);
 extern void scsi_show_extd_sense(const struct scsi_device *, const char *,
 				 unsigned char, unsigned char);
-extern void scsi_show_sense_hdr(const struct scsi_device *, const char *,
-				const struct scsi_sense_hdr *);
 extern void scsi_print_sense_hdr(const struct scsi_device *, const char *,
 				 const struct scsi_sense_hdr *);
 extern void scsi_print_sense(const struct scsi_cmnd *);
-- 
1.8.5.6

[PATCH 08/23] scsi_dh_alua: use flag for RTPG extended header

[Hannes Reinecke]

We should be using a flag when RTPG extended header is not
supported, that saves us sending RTPG twice for older arrays.

Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 25c2045..9d8fc53 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -59,8 +59,9 @@
 #define ALUA_FAILOVER_TIMEOUT		60
 #define ALUA_FAILOVER_RETRIES		5
 
-/* flags passed from user level */
+/* device handler flags */
 #define ALUA_OPTIMIZE_STPG		1
+#define ALUA_RTPG_EXT_HDR_UNSUPP	2
 
 struct alua_dh_data {
 	int			group_id;
@@ -135,8 +136,7 @@ static struct request *get_alua_req(struct scsi_device *sdev,
  * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
  * @sdev: sdev the command should be sent to
  */
-static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
-			    bool rtpg_ext_hdr_req)
+static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
 {
 	struct request *rq;
 	int err = 0;
@@ -149,7 +149,7 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
 
 	/* Prepare the command. */
 	rq->cmd[0] = MAINTENANCE_IN;
-	if (rtpg_ext_hdr_req)
+	if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP))
 		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
 	else
 		rq->cmd[1] = MI_REPORT_TARGET_PGS;
@@ -487,7 +487,6 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 	int len, k, off, valid_states = 0;
 	unsigned char *ucp;
 	unsigned err, retval;
-	bool rtpg_ext_hdr_req = 1;
 	unsigned long expiry, interval = 0;
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
@@ -498,7 +497,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
 
  retry:
-	retval = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
+	retval = submit_rtpg(sdev, h);
 
 	if (retval) {
 		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
@@ -521,10 +520,10 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		 * The retry without rtpg_ext_hdr_req set
 		 * handles this.
 		 */
-		if (rtpg_ext_hdr_req == 1 &&
+		if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
 		    sense_hdr.sense_key == ILLEGAL_REQUEST &&
 		    sense_hdr.asc == 0x24 && sense_hdr.ascq == 0) {
-			rtpg_ext_hdr_req = 0;
+			h->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
 			goto retry;
 		}
 
-- 
1.8.5.6

[PATCH 09/23] scsi_dh_alua: use unaligned access macros

[Hannes Reinecke]

Use 'get_unaligned_XX' and 'put_unaligned_XX' instead of
open-coding it.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 23 ++++++++---------------
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 9d8fc53..0636721 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/delay.h>
 #include <linux/module.h>
+#include <asm/unaligned.h>
 #include <scsi/scsi.h>
 #include <scsi/scsi_dbg.h>
 #include <scsi/scsi_eh.h>
@@ -153,10 +154,7 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
 		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
 	else
 		rq->cmd[1] = MI_REPORT_TARGET_PGS;
-	rq->cmd[6] = (h->bufflen >> 24) & 0xff;
-	rq->cmd[7] = (h->bufflen >> 16) & 0xff;
-	rq->cmd[8] = (h->bufflen >>  8) & 0xff;
-	rq->cmd[9] = h->bufflen & 0xff;
+	put_unaligned_be32(h->bufflen, &rq->cmd[6]);
 	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
 
 	rq->sense = h->sense;
@@ -239,8 +237,7 @@ static unsigned submit_stpg(struct alua_dh_data *h)
 	/* Prepare the data buffer */
 	memset(h->buff, 0, stpg_len);
 	h->buff[4] = TPGS_STATE_OPTIMIZED & 0x0f;
-	h->buff[6] = (h->group_id >> 8) & 0xff;
-	h->buff[7] = h->group_id & 0xff;
+	put_unaligned_be16(h->group_id, &h->buff[6]);
 
 	rq = get_alua_req(sdev, h->buff, stpg_len, WRITE);
 	if (!rq)
@@ -249,10 +246,7 @@ static unsigned submit_stpg(struct alua_dh_data *h)
 	/* Prepare the command. */
 	rq->cmd[0] = MAINTENANCE_OUT;
 	rq->cmd[1] = MO_SET_TARGET_PGS;
-	rq->cmd[6] = (stpg_len >> 24) & 0xff;
-	rq->cmd[7] = (stpg_len >> 16) & 0xff;
-	rq->cmd[8] = (stpg_len >>  8) & 0xff;
-	rq->cmd[9] = stpg_len & 0xff;
+	put_unaligned_be32(stpg_len, &rq->cmd[6]);
 	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
 
 	rq->sense = h->sense;
@@ -341,11 +335,11 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		switch (d[1] & 0xf) {
 		case 0x4:
 			/* Relative target port */
-			h->rel_port = (d[6] << 8) + d[7];
+			h->rel_port = get_unaligned_be16(&d[6]);
 			break;
 		case 0x5:
 			/* Target port group */
-			h->group_id = (d[6] << 8) + d[7];
+			h->group_id = get_unaligned_be16(&d[6]);
 			break;
 		default:
 			break;
@@ -540,8 +534,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		return SCSI_DH_IO;
 	}
 
-	len = (h->buff[0] << 24) + (h->buff[1] << 16) +
-		(h->buff[2] << 8) + h->buff[3] + 4;
+	len = get_unaligned_be32(&h->buff[0]) + 4;
 
 	if (len > h->bufflen) {
 		/* Resubmit with the correct length */
@@ -576,7 +569,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 	     k < len;
 	     k += off, ucp += off) {
 
-		if (h->group_id == (ucp[2] << 8) + ucp[3]) {
+		if (h->group_id == get_unaligned_be16(&ucp[2])) {
 			h->state = ucp[0] & 0x0f;
 			h->pref = ucp[0] >> 7;
 			valid_states = ucp[1];
-- 
1.8.5.6

[PATCH 10/23] scsi_dh_alua: Pass buffer as function argument

[Hannes Reinecke]

Pass in the buffer as a function argument for submit_rtpg().

Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 0636721..9e2b3af 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -137,12 +137,13 @@ static struct request *get_alua_req(struct scsi_device *sdev,
  * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
  * @sdev: sdev the command should be sent to
  */
-static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
+static unsigned submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
+			    int bufflen, unsigned char *sense, int flags)
 {
 	struct request *rq;
 	int err = 0;
 
-	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
+	rq = get_alua_req(sdev, buff, bufflen, READ);
 	if (!rq) {
 		err = DRIVER_BUSY << 24;
 		goto done;
@@ -150,22 +151,21 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
 
 	/* Prepare the command. */
 	rq->cmd[0] = MAINTENANCE_IN;
-	if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP))
+	if (!(flags & ALUA_RTPG_EXT_HDR_UNSUPP))
 		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
 	else
 		rq->cmd[1] = MI_REPORT_TARGET_PGS;
-	put_unaligned_be32(h->bufflen, &rq->cmd[6]);
+	put_unaligned_be32(bufflen, &rq->cmd[6]);
 	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
 
-	rq->sense = h->sense;
+	rq->sense = sense;
 	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
-	rq->sense_len = h->senselen = 0;
+	rq->sense_len = 0;
 
 	blk_execute_rq(rq->q, NULL, rq, 1);
-	if (rq->errors) {
+	if (rq->errors)
 		err = rq->errors;
-		h->senselen = rq->sense_len;
-	}
+
 	blk_put_request(rq);
 done:
 	return err;
@@ -491,7 +491,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
 
  retry:
-	retval = submit_rtpg(sdev, h);
+	retval = submit_rtpg(sdev, h->buff, h->bufflen, h->sense, h->flags);
 
 	if (retval) {
 		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-- 
1.8.5.6

[PATCH 11/23] scsi_dh_alua: Make stpg synchronous

[Hannes Reinecke]

We should be issuing STPG synchronously as we need to
evaluate the return code on failure.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 179 +++++++++++++----------------
 1 file changed, 83 insertions(+), 96 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 9e2b3af..fd0385e 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -172,76 +172,28 @@ done:
 }
 
 /*
- * stpg_endio - Evaluate SET TARGET GROUP STATES
- * @sdev: the device to be evaluated
- * @state: the new target group state
- *
- * Evaluate a SET TARGET GROUP STATES command response.
- */
-static void stpg_endio(struct request *req, int error)
-{
-	struct alua_dh_data *h = req->end_io_data;
-	struct scsi_sense_hdr sense_hdr;
-	unsigned err = SCSI_DH_OK;
-
-	if (host_byte(req->errors) != DID_OK ||
-	    msg_byte(req->errors) != COMMAND_COMPLETE) {
-		err = SCSI_DH_IO;
-		goto done;
-	}
-
-	if (scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-				 &sense_hdr)) {
-		err = alua_check_sense(h->sdev, &sense_hdr);
-		if (err == ADD_TO_MLQUEUE) {
-			err = SCSI_DH_RETRY;
-			goto done;
-		}
-		sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
-			    ALUA_DH_NAME);
-		scsi_print_sense_hdr(h->sdev, ALUA_DH_NAME, &sense_hdr);
-		err = SCSI_DH_IO;
-	} else if (error)
-		err = SCSI_DH_IO;
-
-	if (err == SCSI_DH_OK) {
-		h->state = TPGS_STATE_OPTIMIZED;
-		sdev_printk(KERN_INFO, h->sdev,
-			    "%s: port group %02x switched to state %c\n",
-			    ALUA_DH_NAME, h->group_id,
-			    print_alua_state(h->state));
-	}
-done:
-	req->end_io_data = NULL;
-	__blk_put_request(req->q, req);
-	if (h->callback_fn) {
-		h->callback_fn(h->callback_data, err);
-		h->callback_fn = h->callback_data = NULL;
-	}
-	return;
-}
-
-/*
  * submit_stpg - Issue a SET TARGET GROUP STATES command
  *
  * Currently we're only setting the current target port group state
  * to 'active/optimized' and let the array firmware figure out
  * the states of the remaining groups.
  */
-static unsigned submit_stpg(struct alua_dh_data *h)
+static unsigned submit_stpg(struct scsi_device *sdev, int group_id,
+			    unsigned char *sense)
 {
 	struct request *rq;
+	unsigned char stpg_data[8];
 	int stpg_len = 8;
-	struct scsi_device *sdev = h->sdev;
+	int err = 0;
 
 	/* Prepare the data buffer */
-	memset(h->buff, 0, stpg_len);
-	h->buff[4] = TPGS_STATE_OPTIMIZED & 0x0f;
-	put_unaligned_be16(h->group_id, &h->buff[6]);
+	memset(stpg_data, 0, stpg_len);
+	stpg_data[4] = TPGS_STATE_OPTIMIZED & 0x0f;
+	put_unaligned_be16(group_id, &stpg_data[6]);
 
-	rq = get_alua_req(sdev, h->buff, stpg_len, WRITE);
+	rq = get_alua_req(sdev, stpg_data, stpg_len, WRITE);
 	if (!rq)
-		return SCSI_DH_RES_TEMP_UNAVAIL;
+		return DRIVER_BUSY << 24;
 
 	/* Prepare the command. */
 	rq->cmd[0] = MAINTENANCE_OUT;
@@ -249,13 +201,17 @@ static unsigned submit_stpg(struct alua_dh_data *h)
 	put_unaligned_be32(stpg_len, &rq->cmd[6]);
 	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
 
-	rq->sense = h->sense;
+	rq->sense = sense;
 	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
-	rq->sense_len = h->senselen = 0;
-	rq->end_io_data = h;
+	rq->sense_len = 0;
+
+	blk_execute_rq(rq->q, NULL, rq, 1);
+	if (rq->errors)
+		err = rq->errors;
+
+	blk_put_request(rq);
 
-	blk_execute_rq_nowait(rq->q, NULL, rq, 1, stpg_endio);
-	return SCSI_DH_OK;
+	return err;
 }
 
 /*
@@ -619,6 +575,68 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 }
 
 /*
+ * alua_stpg - Issue a SET TARGET GROUP STATES command
+ *
+ * Issue a SET TARGET GROUP STATES command and evaluate the
+ * response. Returns SCSI_DH_RETRY per default to trigger
+ * a re-evaluation of the target group state.
+ */
+static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
+{
+	int retval;
+	struct scsi_sense_hdr sense_hdr;
+
+	if (!(h->tpgs & TPGS_MODE_EXPLICIT)) {
+		/* Only implicit ALUA supported, retry */
+		return SCSI_DH_RETRY;
+	}
+	switch (h->state) {
+	case TPGS_STATE_OPTIMIZED:
+		return SCSI_DH_OK;
+	case TPGS_STATE_NONOPTIMIZED:
+		if ((h->flags & ALUA_OPTIMIZE_STPG) &&
+		    !h->pref &&
+		    (h->tpgs & TPGS_MODE_IMPLICIT))
+			return SCSI_DH_OK;
+		break;
+	case TPGS_STATE_STANDBY:
+	case TPGS_STATE_UNAVAILABLE:
+		break;
+	case TPGS_STATE_OFFLINE:
+		return SCSI_DH_IO;
+		break;
+	case TPGS_STATE_TRANSITIONING:
+		break;
+	default:
+		sdev_printk(KERN_INFO, sdev,
+			    "%s: stpg failed, unhandled TPGS state %d",
+			    ALUA_DH_NAME, h->state);
+		return SCSI_DH_NOSYS;
+		break;
+	}
+	/* Set state to transitioning */
+	h->state = TPGS_STATE_TRANSITIONING;
+	retval = submit_stpg(sdev, h->group_id, h->sense);
+
+	if (retval) {
+		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
+					  &sense_hdr)) {
+			sdev_printk(KERN_INFO, sdev,
+				    "%s: stpg failed, result %d",
+				    ALUA_DH_NAME, retval);
+			if (driver_byte(retval) == DRIVER_BUSY)
+				return SCSI_DH_DEV_TEMP_BUSY;
+		} else {
+			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
+				    ALUA_DH_NAME);
+			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
+		}
+	}
+	/* Retry RTPG */
+	return SCSI_DH_RETRY;
+}
+
+/*
  * alua_initialize - Initialize ALUA state
  * @sdev: the device to be initialized
  *
@@ -695,7 +713,6 @@ static int alua_activate(struct scsi_device *sdev,
 {
 	struct alua_dh_data *h = sdev->handler_data;
 	int err = SCSI_DH_OK;
-	int stpg = 0;
 
 	err = alua_rtpg(sdev, h, 1);
 	if (err != SCSI_DH_OK)
@@ -704,39 +721,9 @@ static int alua_activate(struct scsi_device *sdev,
 	if (optimize_stpg)
 		h->flags |= ALUA_OPTIMIZE_STPG;
 
-	if (h->tpgs & TPGS_MODE_EXPLICIT) {
-		switch (h->state) {
-		case TPGS_STATE_NONOPTIMIZED:
-			stpg = 1;
-			if ((h->flags & ALUA_OPTIMIZE_STPG) &&
-			    (!h->pref) &&
-			    (h->tpgs & TPGS_MODE_IMPLICIT))
-				stpg = 0;
-			break;
-		case TPGS_STATE_STANDBY:
-		case TPGS_STATE_UNAVAILABLE:
-			stpg = 1;
-			break;
-		case TPGS_STATE_OFFLINE:
-			err = SCSI_DH_IO;
-			break;
-		case TPGS_STATE_TRANSITIONING:
-			err = SCSI_DH_RETRY;
-			break;
-		default:
-			break;
-		}
-	}
-
-	if (stpg) {
-		h->callback_fn = fn;
-		h->callback_data = data;
-		err = submit_stpg(h);
-		if (err == SCSI_DH_OK)
-			return 0;
-		h->callback_fn = h->callback_data = NULL;
-	}
-
+	err = alua_stpg(sdev, h);
+	if (err == SCSI_DH_RETRY)
+		err = alua_rtpg(sdev, h, 1);
 out:
 	if (fn)
 		fn(data, err);
-- 
1.8.5.6

[PATCH 12/23] scsi_dh_alua: switch to scsi_execute_req_flags()

[Hannes Reinecke]

All commands are issued synchronously, so no need to open-code
scsi_execute_req_flags() anymore. And we can get rid of the
static sense code structure element.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 124 ++++++++---------------------
 1 file changed, 34 insertions(+), 90 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index fd0385e..ef4363a 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -75,8 +75,6 @@ struct alua_dh_data {
 	unsigned char		*buff;
 	int			bufflen;
 	unsigned char		transition_tmo;
-	unsigned char		sense[SCSI_SENSE_BUFFERSIZE];
-	int			senselen;
 	struct scsi_device	*sdev;
 	activate_complete	callback_fn;
 	void			*callback_data;
@@ -103,72 +101,30 @@ static int realloc_buffer(struct alua_dh_data *h, unsigned len)
 	return 0;
 }
 
-static struct request *get_alua_req(struct scsi_device *sdev,
-				    void *buffer, unsigned buflen, int rw)
-{
-	struct request *rq;
-	struct request_queue *q = sdev->request_queue;
-
-	rq = blk_get_request(q, rw, GFP_NOIO);
-
-	if (IS_ERR(rq)) {
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: blk_get_request failed\n", __func__);
-		return NULL;
-	}
-	blk_rq_set_block_pc(rq);
-
-	if (buflen && blk_rq_map_kern(q, rq, buffer, buflen, GFP_NOIO)) {
-		blk_put_request(rq);
-		sdev_printk(KERN_INFO, sdev,
-			    "%s: blk_rq_map_kern failed\n", __func__);
-		return NULL;
-	}
-
-	rq->cmd_flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
-			 REQ_FAILFAST_DRIVER;
-	rq->retries = ALUA_FAILOVER_RETRIES;
-	rq->timeout = ALUA_FAILOVER_TIMEOUT * HZ;
-
-	return rq;
-}
-
 /*
  * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
  * @sdev: sdev the command should be sent to
  */
-static unsigned submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
-			    int bufflen, unsigned char *sense, int flags)
+static int submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
+		       int bufflen, struct scsi_sense_hdr *sshdr, int flags)
 {
-	struct request *rq;
-	int err = 0;
-
-	rq = get_alua_req(sdev, buff, bufflen, READ);
-	if (!rq) {
-		err = DRIVER_BUSY << 24;
-		goto done;
-	}
+	u8 cdb[16];
+	int req_flags = REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
+		REQ_FAILFAST_DRIVER;
 
 	/* Prepare the command. */
-	rq->cmd[0] = MAINTENANCE_IN;
+	memset(cdb, 0x0, 16);
+	cdb[0] = MAINTENANCE_IN;
 	if (!(flags & ALUA_RTPG_EXT_HDR_UNSUPP))
-		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
+		cdb[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
 	else
-		rq->cmd[1] = MI_REPORT_TARGET_PGS;
-	put_unaligned_be32(bufflen, &rq->cmd[6]);
-	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
-
-	rq->sense = sense;
-	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
-	rq->sense_len = 0;
+		cdb[1] = MI_REPORT_TARGET_PGS;
+	put_unaligned_be32(bufflen, &cdb[6]);
 
-	blk_execute_rq(rq->q, NULL, rq, 1);
-	if (rq->errors)
-		err = rq->errors;
-
-	blk_put_request(rq);
-done:
-	return err;
+	return scsi_execute_req_flags(sdev, cdb, DMA_FROM_DEVICE,
+				      buff, bufflen, sshdr,
+				      ALUA_FAILOVER_TIMEOUT * HZ,
+				      ALUA_FAILOVER_RETRIES, NULL, req_flags);
 }
 
 /*
@@ -178,40 +134,30 @@ done:
  * to 'active/optimized' and let the array firmware figure out
  * the states of the remaining groups.
  */
-static unsigned submit_stpg(struct scsi_device *sdev, int group_id,
-			    unsigned char *sense)
+static int submit_stpg(struct scsi_device *sdev, int group_id,
+		       struct scsi_sense_hdr *sshdr)
 {
-	struct request *rq;
+	u8 cdb[COMMAND_SIZE(MAINTENANCE_OUT)];
 	unsigned char stpg_data[8];
 	int stpg_len = 8;
-	int err = 0;
+	int req_flags = REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
+		REQ_FAILFAST_DRIVER;
 
 	/* Prepare the data buffer */
 	memset(stpg_data, 0, stpg_len);
 	stpg_data[4] = TPGS_STATE_OPTIMIZED & 0x0f;
 	put_unaligned_be16(group_id, &stpg_data[6]);
 
-	rq = get_alua_req(sdev, stpg_data, stpg_len, WRITE);
-	if (!rq)
-		return DRIVER_BUSY << 24;
-
 	/* Prepare the command. */
-	rq->cmd[0] = MAINTENANCE_OUT;
-	rq->cmd[1] = MO_SET_TARGET_PGS;
-	put_unaligned_be32(stpg_len, &rq->cmd[6]);
-	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
-
-	rq->sense = sense;
-	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
-	rq->sense_len = 0;
-
-	blk_execute_rq(rq->q, NULL, rq, 1);
-	if (rq->errors)
-		err = rq->errors;
-
-	blk_put_request(rq);
-
-	return err;
+	memset(cdb, 0x0, COMMAND_SIZE(MAINTENANCE_OUT));
+	cdb[0] = MAINTENANCE_OUT;
+	cdb[1] = MO_SET_TARGET_PGS;
+	put_unaligned_be32(stpg_len, &cdb[6]);
+
+	return scsi_execute_req_flags(sdev, cdb, DMA_TO_DEVICE,
+				      stpg_data, stpg_len,
+				      sshdr, ALUA_FAILOVER_TIMEOUT * HZ,
+				      ALUA_FAILOVER_RETRIES, NULL, req_flags);
 }
 
 /*
@@ -447,15 +393,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
 
  retry:
-	retval = submit_rtpg(sdev, h->buff, h->bufflen, h->sense, h->flags);
+	retval = submit_rtpg(sdev, h->buff, h->bufflen, &sense_hdr, h->flags);
 
 	if (retval) {
-		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-					  &sense_hdr)) {
+		if (!scsi_sense_valid(&sense_hdr)) {
 			sdev_printk(KERN_INFO, sdev,
 				    "%s: rtpg failed, result %d\n",
 				    ALUA_DH_NAME, retval);
-			if (driver_byte(retval) == DRIVER_BUSY)
+			if (driver_byte(retval) == DRIVER_ERROR)
 				err = SCSI_DH_DEV_TEMP_BUSY;
 			else
 				err = SCSI_DH_IO;
@@ -616,15 +561,14 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
 	}
 	/* Set state to transitioning */
 	h->state = TPGS_STATE_TRANSITIONING;
-	retval = submit_stpg(sdev, h->group_id, h->sense);
+	retval = submit_stpg(sdev, h->group_id, &sense_hdr);
 
 	if (retval) {
-		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
-					  &sense_hdr)) {
+		if (!scsi_sense_valid(&sense_hdr)) {
 			sdev_printk(KERN_INFO, sdev,
 				    "%s: stpg failed, result %d",
 				    ALUA_DH_NAME, retval);
-			if (driver_byte(retval) == DRIVER_BUSY)
+			if (driver_byte(retval) == DRIVER_ERROR)
 				return SCSI_DH_DEV_TEMP_BUSY;
 		} else {
 			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
-- 
1.8.5.6

[PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Hannes Reinecke]

The port group needs to be a separate structure as several
LUNs might belong to the same group.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 211 +++++++++++++++++++----------
 1 file changed, 139 insertions(+), 72 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index ef4363a..d1010dd 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -64,9 +64,13 @@
 #define ALUA_OPTIMIZE_STPG		1
 #define ALUA_RTPG_EXT_HDR_UNSUPP	2
 
-struct alua_dh_data {
+static LIST_HEAD(port_group_list);
+static DEFINE_SPINLOCK(port_group_lock);
+
+struct alua_port_group {
+	struct kref		kref;
+	struct list_head	node;
 	int			group_id;
-	int			rel_port;
 	int			tpgs;
 	int			state;
 	int			pref;
@@ -75,6 +79,12 @@ struct alua_dh_data {
 	unsigned char		*buff;
 	int			bufflen;
 	unsigned char		transition_tmo;
+};
+
+struct alua_dh_data {
+	struct alua_port_group	*pg;
+	int			rel_port;
+	int			tpgs;
 	struct scsi_device	*sdev;
 	activate_complete	callback_fn;
 	void			*callback_data;
@@ -86,21 +96,35 @@ struct alua_dh_data {
 static char print_alua_state(int);
 static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
 
-static int realloc_buffer(struct alua_dh_data *h, unsigned len)
+static int realloc_buffer(struct alua_port_group *pg, unsigned len)
 {
-	if (h->buff && h->buff != h->inq)
-		kfree(h->buff);
+	if (pg->buff && pg->buff != pg->inq)
+		kfree(pg->buff);
 
-	h->buff = kmalloc(len, GFP_NOIO);
-	if (!h->buff) {
-		h->buff = h->inq;
-		h->bufflen = ALUA_INQUIRY_SIZE;
+	pg->buff = kmalloc(len, GFP_NOIO);
+	if (!pg->buff) {
+		pg->buff = pg->inq;
+		pg->bufflen = ALUA_INQUIRY_SIZE;
 		return 1;
 	}
-	h->bufflen = len;
+	pg->bufflen = len;
 	return 0;
 }
 
+static void release_port_group(struct kref *kref)
+{
+	struct alua_port_group *pg;
+
+	pg = container_of(kref, struct alua_port_group, kref);
+	printk(KERN_WARNING "alua: release port group %d\n", pg->group_id);
+	spin_lock(&port_group_lock);
+	list_del(&pg->node);
+	spin_unlock(&port_group_lock);
+	if (pg->buff && pg->inq != pg->buff)
+		kfree(pg->buff);
+	kfree(pg);
+}
+
 /*
  * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
  * @sdev: sdev the command should be sent to
@@ -225,6 +249,8 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
 static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 {
 	unsigned char *d;
+	int group_id = -1;
+	struct alua_port_group *pg = NULL;
 
 	if (!sdev->vpd_pg83)
 		return SCSI_DH_DEV_UNSUPP;
@@ -241,7 +267,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 			break;
 		case 0x5:
 			/* Target port group */
-			h->group_id = get_unaligned_be16(&d[6]);
+			group_id = get_unaligned_be16(&d[6]);
 			break;
 		default:
 			break;
@@ -249,7 +275,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		d += d[3] + 4;
 	}
 
-	if (h->group_id == -1) {
+	if (group_id == -1) {
 		/*
 		 * Internal error; TPGS supported but required
 		 * VPD identification descriptors not present.
@@ -258,15 +284,33 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		sdev_printk(KERN_INFO, sdev,
 			    "%s: No target port descriptors found\n",
 			    ALUA_DH_NAME);
-		h->state = TPGS_STATE_OPTIMIZED;
 		h->tpgs = TPGS_MODE_NONE;
 		return SCSI_DH_DEV_UNSUPP;
 	}
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x rel port %02x\n",
-		    ALUA_DH_NAME, h->group_id, h->rel_port);
+		    ALUA_DH_NAME, group_id, h->rel_port);
 
-	return 0;
+	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
+	if (!pg) {
+		sdev_printk(KERN_WARNING, sdev,
+			    "%s: kzalloc port group failed\n",
+			    ALUA_DH_NAME);
+		/* Temporary failure, bypass */
+		return SCSI_DH_DEV_TEMP_BUSY;
+	}
+	pg->group_id = group_id;
+	pg->buff = pg->inq;
+	pg->bufflen = ALUA_INQUIRY_SIZE;
+	pg->tpgs = h->tpgs;
+	pg->state = TPGS_STATE_OPTIMIZED;
+	kref_init(&pg->kref);
+	spin_lock(&port_group_lock);
+	list_add(&pg->node, &port_group_list);
+	h->pg = pg;
+	spin_unlock(&port_group_lock);
+
+	return SCSI_DH_OK;
 }
 
 static char print_alua_state(int state)
@@ -377,7 +421,7 @@ static int alua_check_sense(struct scsi_device *sdev,
  * Returns SCSI_DH_DEV_OFFLINED if the path is
  * found to be unusable.
  */
-static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_for_transition)
+static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
 {
 	struct scsi_sense_hdr sense_hdr;
 	int len, k, off, valid_states = 0;
@@ -387,13 +431,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 
-	if (!h->transition_tmo)
+	if (!pg->transition_tmo)
 		expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
 	else
-		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
+		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
 
  retry:
-	retval = submit_rtpg(sdev, h->buff, h->bufflen, &sense_hdr, h->flags);
+	retval = submit_rtpg(sdev, pg->buff, pg->bufflen,
+			     &sense_hdr, pg->flags);
 
 	if (retval) {
 		if (!scsi_sense_valid(&sense_hdr)) {
@@ -415,10 +460,10 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		 * The retry without rtpg_ext_hdr_req set
 		 * handles this.
 		 */
-		if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
+		if (!(pg->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
 		    sense_hdr.sense_key == ILLEGAL_REQUEST &&
 		    sense_hdr.asc == 0x24 && sense_hdr.ascq == 0) {
-			h->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
+			pg->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
 			goto retry;
 		}
 
@@ -435,11 +480,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		return SCSI_DH_IO;
 	}
 
-	len = get_unaligned_be32(&h->buff[0]) + 4;
+	len = get_unaligned_be32(&pg->buff[0]) + 4;
 
-	if (len > h->bufflen) {
+	if (len > pg->bufflen) {
 		/* Resubmit with the correct length */
-		if (realloc_buffer(h, len)) {
+		if (realloc_buffer(pg, len)) {
 			sdev_printk(KERN_WARNING, sdev,
 				    "%s: kmalloc buffer failed\n",__func__);
 			/* Temporary failure, bypass */
@@ -448,31 +493,33 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		goto retry;
 	}
 
-	orig_transition_tmo = h->transition_tmo;
-	if ((h->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR && h->buff[5] != 0)
-		h->transition_tmo = h->buff[5];
+	orig_transition_tmo = pg->transition_tmo;
+	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR &&
+	    pg->buff[5] != 0)
+		pg->transition_tmo = pg->buff[5];
 	else
-		h->transition_tmo = ALUA_FAILOVER_TIMEOUT;
+		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
 
-	if (wait_for_transition && (orig_transition_tmo != h->transition_tmo)) {
+	if (wait_for_transition &&
+	    (orig_transition_tmo != pg->transition_tmo)) {
 		sdev_printk(KERN_INFO, sdev,
 			    "%s: transition timeout set to %d seconds\n",
-			    ALUA_DH_NAME, h->transition_tmo);
-		expiry = jiffies + h->transition_tmo * HZ;
+			    ALUA_DH_NAME, pg->transition_tmo);
+		expiry = jiffies + pg->transition_tmo * HZ;
 	}
 
-	if ((h->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
+	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
 		tpg_desc_tbl_off = 8;
 	else
 		tpg_desc_tbl_off = 4;
 
-	for (k = tpg_desc_tbl_off, ucp = h->buff + tpg_desc_tbl_off;
+	for (k = tpg_desc_tbl_off, ucp = pg->buff + tpg_desc_tbl_off;
 	     k < len;
 	     k += off, ucp += off) {
 
-		if (h->group_id == get_unaligned_be16(&ucp[2])) {
-			h->state = ucp[0] & 0x0f;
-			h->pref = ucp[0] >> 7;
+		if (pg->group_id == get_unaligned_be16(&ucp[2])) {
+			pg->state = ucp[0] & 0x0f;
+			pg->pref = ucp[0] >> 7;
 			valid_states = ucp[1];
 		}
 		off = 8 + (ucp[7] * 4);
@@ -480,8 +527,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
-		    ALUA_DH_NAME, h->group_id, print_alua_state(h->state),
-		    h->pref ? "preferred" : "non-preferred",
+		    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),
+		    pg->pref ? "preferred" : "non-preferred",
 		    valid_states&TPGS_SUPPORT_TRANSITION?'T':'t',
 		    valid_states&TPGS_SUPPORT_OFFLINE?'O':'o',
 		    valid_states&TPGS_SUPPORT_LBA_DEPENDENT?'L':'l',
@@ -490,7 +537,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		    valid_states&TPGS_SUPPORT_NONOPTIMIZED?'N':'n',
 		    valid_states&TPGS_SUPPORT_OPTIMIZED?'A':'a');
 
-	switch (h->state) {
+	switch (pg->state) {
 	case TPGS_STATE_TRANSITIONING:
 		if (wait_for_transition) {
 			if (time_before(jiffies, expiry)) {
@@ -505,7 +552,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
 		}
 
 		/* Transitioning time exceeded, set port to standby */
-		h->state = TPGS_STATE_STANDBY;
+		pg->state = TPGS_STATE_STANDBY;
 		break;
 	case TPGS_STATE_OFFLINE:
 		/* Path unusable */
@@ -526,22 +573,22 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
  * response. Returns SCSI_DH_RETRY per default to trigger
  * a re-evaluation of the target group state.
  */
-static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
+static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
 {
 	int retval;
 	struct scsi_sense_hdr sense_hdr;
 
-	if (!(h->tpgs & TPGS_MODE_EXPLICIT)) {
+	if (!(pg->tpgs & TPGS_MODE_EXPLICIT)) {
 		/* Only implicit ALUA supported, retry */
 		return SCSI_DH_RETRY;
 	}
-	switch (h->state) {
+	switch (pg->state) {
 	case TPGS_STATE_OPTIMIZED:
 		return SCSI_DH_OK;
 	case TPGS_STATE_NONOPTIMIZED:
-		if ((h->flags & ALUA_OPTIMIZE_STPG) &&
-		    !h->pref &&
-		    (h->tpgs & TPGS_MODE_IMPLICIT))
+		if ((pg->flags & ALUA_OPTIMIZE_STPG) &&
+		    !pg->pref &&
+		    (pg->tpgs & TPGS_MODE_IMPLICIT))
 			return SCSI_DH_OK;
 		break;
 	case TPGS_STATE_STANDBY:
@@ -555,13 +602,13 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
 	default:
 		sdev_printk(KERN_INFO, sdev,
 			    "%s: stpg failed, unhandled TPGS state %d",
-			    ALUA_DH_NAME, h->state);
+			    ALUA_DH_NAME, pg->state);
 		return SCSI_DH_NOSYS;
 		break;
 	}
 	/* Set state to transitioning */
-	h->state = TPGS_STATE_TRANSITIONING;
-	retval = submit_stpg(sdev, h->group_id, &sense_hdr);
+	pg->state = TPGS_STATE_TRANSITIONING;
+	retval = submit_stpg(sdev, pg->group_id, &sense_hdr);
 
 	if (retval) {
 		if (!scsi_sense_valid(&sense_hdr)) {
@@ -571,7 +618,7 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
 			if (driver_byte(retval) == DRIVER_ERROR)
 				return SCSI_DH_DEV_TEMP_BUSY;
 		} else {
-			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
+			sdev_printk(KERN_INFO, sdev, "%s: stpg failed\n",
 				    ALUA_DH_NAME);
 			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
 		}
@@ -596,13 +643,12 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
 		goto out;
 
 	err = alua_check_vpd(sdev, h);
-	if (err != SCSI_DH_OK)
-		goto out;
-
-	err = alua_rtpg(sdev, h, 0);
-	if (err != SCSI_DH_OK)
+	if (err != SCSI_DH_OK || !h->pg)
 		goto out;
 
+	kref_get(&h->pg->kref);
+	err = alua_rtpg(sdev, h->pg, 0);
+	kref_put(&h->pg->kref, release_port_group);
 out:
 	return err;
 }
@@ -618,6 +664,7 @@ out:
 static int alua_set_params(struct scsi_device *sdev, const char *params)
 {
 	struct alua_dh_data *h = sdev->handler_data;
+	struct alua_port_group *pg = NULL;
 	unsigned int optimize = 0, argc;
 	const char *p = params;
 	int result = SCSI_DH_OK;
@@ -630,10 +677,18 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
 	if ((sscanf(p, "%u", &optimize) != 1) || (optimize > 1))
 		return -EINVAL;
 
+	rcu_read_lock();
+	pg = rcu_dereference(h->pg);
+	if (!pg) {
+		rcu_read_unlock();
+		return -ENXIO;
+	}
+
 	if (optimize)
-		h->flags |= ALUA_OPTIMIZE_STPG;
+		pg->flags |= ALUA_OPTIMIZE_STPG;
 	else
-		h->flags &= ~ALUA_OPTIMIZE_STPG;
+		pg->flags |= ~ALUA_OPTIMIZE_STPG;
+	rcu_read_unlock();
 
 	return result;
 }
@@ -658,16 +713,23 @@ static int alua_activate(struct scsi_device *sdev,
 	struct alua_dh_data *h = sdev->handler_data;
 	int err = SCSI_DH_OK;
 
-	err = alua_rtpg(sdev, h, 1);
-	if (err != SCSI_DH_OK)
+	if (!h->pg)
 		goto out;
 
+	kref_get(&h->pg->kref);
+
 	if (optimize_stpg)
-		h->flags |= ALUA_OPTIMIZE_STPG;
+		h->pg->flags |= ALUA_OPTIMIZE_STPG;
 
-	err = alua_stpg(sdev, h);
+	err = alua_rtpg(sdev, h->pg, 1);
+	if (err != SCSI_DH_OK) {
+		kref_put(&h->pg->kref, release_port_group);
+		goto out;
+	}
+	err = alua_stpg(sdev, h->pg);
 	if (err == SCSI_DH_RETRY)
-		err = alua_rtpg(sdev, h, 1);
+		err = alua_rtpg(sdev, h->pg, 1);
+	kref_put(&h->pg->kref, release_port_group);
 out:
 	if (fn)
 		fn(data, err);
@@ -683,13 +745,19 @@ out:
 static int alua_prep_fn(struct scsi_device *sdev, struct request *req)
 {
 	struct alua_dh_data *h = sdev->handler_data;
+	int state;
 	int ret = BLKPREP_OK;
 
-	if (h->state == TPGS_STATE_TRANSITIONING)
+	if (!h->pg)
+		return ret;
+	kref_get(&h->pg->kref);
+	state = h->pg->state;
+	kref_put(&h->pg->kref, release_port_group);
+	if (state == TPGS_STATE_TRANSITIONING)
 		ret = BLKPREP_DEFER;
-	else if (h->state != TPGS_STATE_OPTIMIZED &&
-		 h->state != TPGS_STATE_NONOPTIMIZED &&
-		 h->state != TPGS_STATE_LBA_DEPENDENT) {
+	else if (state != TPGS_STATE_OPTIMIZED &&
+		 state != TPGS_STATE_NONOPTIMIZED &&
+		 state != TPGS_STATE_LBA_DEPENDENT) {
 		ret = BLKPREP_KILL;
 		req->cmd_flags |= REQ_QUIET;
 	}
@@ -710,11 +778,8 @@ static int alua_bus_attach(struct scsi_device *sdev)
 	if (!h)
 		return -ENOMEM;
 	h->tpgs = TPGS_MODE_UNINITIALIZED;
-	h->state = TPGS_STATE_OPTIMIZED;
-	h->group_id = -1;
+	h->pg = NULL;
 	h->rel_port = -1;
-	h->buff = h->inq;
-	h->bufflen = ALUA_INQUIRY_SIZE;
 	h->sdev = sdev;
 
 	err = alua_initialize(sdev, h);
@@ -736,8 +801,10 @@ static void alua_bus_detach(struct scsi_device *sdev)
 {
 	struct alua_dh_data *h = sdev->handler_data;
 
-	if (h->buff && h->inq != h->buff)
-		kfree(h->buff);
+	if (h->pg) {
+		kref_put(&h->pg->kref, release_port_group);
+		h->pg = NULL;
+	}
 	sdev->handler_data = NULL;
 	kfree(h);
 }
-- 
1.8.5.6

[PATCH 14/23] scsi_dh_alua: allocate RTPG buffer separately

[Hannes Reinecke]

The RTPG buffer will only evaluated within alua_rtpg(),
so we can allocate it locally there and avoid having to
put it into the global structure.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 56 +++++++++++-------------------
 1 file changed, 21 insertions(+), 35 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index d1010dd..4157fe2 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -56,7 +56,7 @@
 #define TPGS_MODE_IMPLICIT		0x1
 #define TPGS_MODE_EXPLICIT		0x2
 
-#define ALUA_INQUIRY_SIZE		36
+#define ALUA_RTPG_SIZE			128
 #define ALUA_FAILOVER_TIMEOUT		60
 #define ALUA_FAILOVER_RETRIES		5
 
@@ -75,9 +75,6 @@ struct alua_port_group {
 	int			state;
 	int			pref;
 	unsigned		flags; /* used for optimizing STPG */
-	unsigned char		inq[ALUA_INQUIRY_SIZE];
-	unsigned char		*buff;
-	int			bufflen;
 	unsigned char		transition_tmo;
 };
 
@@ -96,21 +93,6 @@ struct alua_dh_data {
 static char print_alua_state(int);
 static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
 
-static int realloc_buffer(struct alua_port_group *pg, unsigned len)
-{
-	if (pg->buff && pg->buff != pg->inq)
-		kfree(pg->buff);
-
-	pg->buff = kmalloc(len, GFP_NOIO);
-	if (!pg->buff) {
-		pg->buff = pg->inq;
-		pg->bufflen = ALUA_INQUIRY_SIZE;
-		return 1;
-	}
-	pg->bufflen = len;
-	return 0;
-}
-
 static void release_port_group(struct kref *kref)
 {
 	struct alua_port_group *pg;
@@ -120,8 +102,6 @@ static void release_port_group(struct kref *kref)
 	spin_lock(&port_group_lock);
 	list_del(&pg->node);
 	spin_unlock(&port_group_lock);
-	if (pg->buff && pg->inq != pg->buff)
-		kfree(pg->buff);
 	kfree(pg);
 }
 
@@ -300,8 +280,6 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		return SCSI_DH_DEV_TEMP_BUSY;
 	}
 	pg->group_id = group_id;
-	pg->buff = pg->inq;
-	pg->bufflen = ALUA_INQUIRY_SIZE;
 	pg->tpgs = h->tpgs;
 	pg->state = TPGS_STATE_OPTIMIZED;
 	kref_init(&pg->kref);
@@ -424,8 +402,8 @@ static int alua_check_sense(struct scsi_device *sdev,
 static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
 {
 	struct scsi_sense_hdr sense_hdr;
-	int len, k, off, valid_states = 0;
-	unsigned char *ucp;
+	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
+	unsigned char *ucp, *buff;
 	unsigned err, retval;
 	unsigned long expiry, interval = 0;
 	unsigned int tpg_desc_tbl_off;
@@ -436,9 +414,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 	else
 		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
 
+	buff = kzalloc(bufflen, GFP_KERNEL);
+	if (!buff)
+		return SCSI_DH_DEV_TEMP_BUSY;
+
  retry:
-	retval = submit_rtpg(sdev, pg->buff, pg->bufflen,
-			     &sense_hdr, pg->flags);
+	retval = submit_rtpg(sdev, buff, bufflen, &sense_hdr, pg->flags);
 
 	if (retval) {
 		if (!scsi_sense_valid(&sense_hdr)) {
@@ -449,6 +430,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 				err = SCSI_DH_DEV_TEMP_BUSY;
 			else
 				err = SCSI_DH_IO;
+			kfree(buff);
 			return err;
 		}
 
@@ -477,14 +459,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
 			    ALUA_DH_NAME);
 		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
+		kfree(buff);
 		return SCSI_DH_IO;
 	}
 
-	len = get_unaligned_be32(&pg->buff[0]) + 4;
+	len = get_unaligned_be32(&buff[0]) + 4;
 
-	if (len > pg->bufflen) {
+	if (len > bufflen) {
 		/* Resubmit with the correct length */
-		if (realloc_buffer(pg, len)) {
+		kfree(buff);
+		bufflen = len;
+		buff = kmalloc(bufflen, GFP_KERNEL);
+		if (!buff) {
 			sdev_printk(KERN_WARNING, sdev,
 				    "%s: kmalloc buffer failed\n",__func__);
 			/* Temporary failure, bypass */
@@ -494,9 +480,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 	}
 
 	orig_transition_tmo = pg->transition_tmo;
-	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR &&
-	    pg->buff[5] != 0)
-		pg->transition_tmo = pg->buff[5];
+	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR && buff[5] != 0)
+		pg->transition_tmo = buff[5];
 	else
 		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
 
@@ -508,12 +493,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 		expiry = jiffies + pg->transition_tmo * HZ;
 	}
 
-	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
+	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
 		tpg_desc_tbl_off = 8;
 	else
 		tpg_desc_tbl_off = 4;
 
-	for (k = tpg_desc_tbl_off, ucp = pg->buff + tpg_desc_tbl_off;
+	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
 	     k < len;
 	     k += off, ucp += off) {
 
@@ -563,6 +548,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 		err = SCSI_DH_OK;
 		break;
 	}
+	kfree(buff);
 	return err;
 }
 
-- 
1.8.5.6

[PATCH 15/23] scsi_dh_alua: simplify sense code handling

[Hannes Reinecke]

Most sense code is already handled in the generic
code, so we shouldn't be adding special cases here.
However, when doing so we need to check for
unit attention whenever we're sending an internal
command.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 50 +++++++-----------------------
 1 file changed, 11 insertions(+), 39 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 4157fe2..dbe9ff2 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -91,7 +91,6 @@ struct alua_dh_data {
 #define ALUA_POLICY_SWITCH_ALL		1
 
 static char print_alua_state(int);
-static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
 
 static void release_port_group(struct kref *kref)
 {
@@ -323,28 +322,6 @@ static int alua_check_sense(struct scsi_device *sdev,
 			 * LUN Not Accessible - ALUA state transition
 			 */
 			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0b)
-			/*
-			 * LUN Not Accessible -- Target port in standby state
-			 */
-			return SUCCESS;
-		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0c)
-			/*
-			 * LUN Not Accessible -- Target port in unavailable state
-			 */
-			return SUCCESS;
-		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x12)
-			/*
-			 * LUN Not Ready -- Offline
-			 */
-			return SUCCESS;
-		if (sdev->allow_restart &&
-		    sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x02)
-			/*
-			 * if the device is not started, we need to wake
-			 * the error handler to start the motor
-			 */
-			return FAILED;
 		break;
 	case UNIT_ATTENTION:
 		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
@@ -359,7 +336,7 @@ static int alua_check_sense(struct scsi_device *sdev,
 			return ADD_TO_MLQUEUE;
 		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
 			/*
-			 * Mode Parameters Changed
+			 * Mode Parameter Changed
 			 */
 			return ADD_TO_MLQUEUE;
 		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
@@ -372,18 +349,6 @@ static int alua_check_sense(struct scsi_device *sdev,
 			 * Implicit ALUA state transition failed
 			 */
 			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x03)
-			/*
-			 * Inquiry data has changed
-			 */
-			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x0e)
-			/*
-			 * REPORTED_LUNS_DATA_HAS_CHANGED is reported
-			 * when switching controllers on targets like
-			 * Intel Multi-Flex. We can just retry.
-			 */
-			return ADD_TO_MLQUEUE;
 		break;
 	}
 
@@ -448,9 +413,16 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 			pg->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
 			goto retry;
 		}
-
-		err = alua_check_sense(sdev, &sense_hdr);
-		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
+		/*
+		 * Retry on ALUA state transition or if any
+		 * UNIT ATTENTION occurred.
+		 */
+		if (sense_hdr.sense_key == NOT_READY &&
+		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
+			err = SCSI_DH_RETRY;
+		else if (sense_hdr.sense_key == UNIT_ATTENTION)
+			err = SCSI_DH_RETRY;
+		if (err == SCSI_DH_RETRY && time_before(jiffies, expiry)) {
 			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
 				    ALUA_DH_NAME);
 			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
-- 
1.8.5.6

[PATCH 16/23] scsi: Add scsi_vpd_lun_id()

[Hannes Reinecke]

Add a function scsi_vpd_lun_id() to return a unique device
identifcation based on the designation descriptors of
VPD page 0x83.

As devices might implement several descriptors the order
of preference is:
- NAA IEE Registered Extended
- EUI-64 based 16-byte
- EUI-64 based 12-byte
- NAA IEEE Registered
- NAA IEEE Extended
A SCSI name string descriptor is preferred to all of them
if the identification is longer than 16 bytes.

The returned unique device identification will be formatted
as a SCSI Name string to avoid clashes between different
designator types.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/scsi_lib.c    | 121 +++++++++++++++++++++++++++++++++++++++++++++
 include/scsi/scsi_device.h |   1 +
 2 files changed, 122 insertions(+)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 2a864b8..48d6ff6 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -3145,3 +3145,124 @@ void sdev_enable_disk_events(struct scsi_device *sdev)
 	atomic_dec(&sdev->disk_events_disable_depth);
 }
 EXPORT_SYMBOL(sdev_enable_disk_events);
+
+/*
+ * scsi_vpd_lun_id - return a unique device identification
+ * @sdev: SCSI device
+ * @id:   buffer for the identification
+ * @id_len:  length of the buffer
+ *
+ * Copies a unique device identification into @id based
+ * on the information in the VPD page 0x83 of the device.
+ * The string will be formatted as a SCSI name string.
+ *
+ * Returns the length of the identification or error on failure.
+ */
+int scsi_vpd_lun_id(struct scsi_device *sdev, char *id, size_t id_len)
+{
+	u8 cur_id_type = 0xff;
+	u8 cur_id_size = 0;
+	unsigned char *d, *cur_id_str;
+	int id_size = -EAGAIN;
+
+	/*
+	 * Look for the correct descriptor.
+	 * Order of preference for lun descriptor:
+	 * - SCSI name string
+	 * - NAA IEEE Registered Extended
+	 * - EUI-64 based 16-byte
+	 * - EUI-64 based 12-byte
+	 * - NAA IEEE Registered
+	 * - NAA IEEE Extended
+	 * as longer descriptors reduce the likelyhood
+	 * of identification clashes.
+	 */
+
+	/* The id string must be at least 20 bytes + terminating NULL byte */
+	if (id_len < 21)
+		return -EINVAL;
+
+	memset(id, 0, id_len);
+	d = sdev->vpd_pg83 + 4;
+	while (d < sdev->vpd_pg83 + sdev->vpd_pg83_len) {
+		/* Skip designators not referring to the LUN */
+		if ((d[1] & 0x30) != 0x00)
+			goto next_desig;
+
+		switch (d[1] & 0xf) {
+		case 0x2:
+			/* EUI-64 */
+			if (cur_id_size > d[3])
+				break;
+			/* Prefer NAA IEEE Registered Extended */
+			if (cur_id_type == 0x3 &&
+			    cur_id_size == d[3])
+				break;
+			cur_id_size = d[3];
+			cur_id_str = d + 4;
+			cur_id_type = d[1] & 0xf;
+			switch (cur_id_size) {
+			case 8:
+				id_size = snprintf(id, id_len,
+						   "eui.%8phN",
+						   cur_id_str);
+				break;
+			case 12:
+				id_size = snprintf(id, id_len,
+						   "eui.%12phN",
+						   cur_id_str);
+				break;
+			case 16:
+				id_size = snprintf(id, id_len,
+						   "eui.%16phN",
+						   cur_id_str);
+				break;
+			default:
+				cur_id_size = 0;
+				break;
+			}
+			break;
+		case 0x3:
+			/* NAA */
+			if (cur_id_size > d[3])
+				break;
+			cur_id_size = d[3];
+			cur_id_str = d + 4;
+			cur_id_type = d[1] & 0xf;
+			switch (cur_id_size) {
+			case 8:
+				id_size = snprintf(id, id_len,
+						   "naa.%8phN",
+						   cur_id_str);
+				break;
+			case 16:
+				id_size = snprintf(id, id_len,
+						   "naa.%16phN",
+						   cur_id_str);
+				break;
+			default:
+				cur_id_size = 0;
+				break;
+			}
+			break;
+		case 0x8:
+			/* SCSI name string */
+			if (cur_id_size + 4 > d[3])
+				break;
+			cur_id_size = d[3];
+			cur_id_str = d + 4;
+			cur_id_type = d[1] & 0xf;
+			if (cur_id_size >= id_len)
+				cur_id_size = id_len - 1;
+			memcpy(id, cur_id_str, cur_id_size);
+			break;
+		default:
+			break;
+		}
+next_desig:
+		d += d[3] + 4;
+	}
+
+	return id_size;
+}
+EXPORT_SYMBOL(scsi_vpd_lun_id);
diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
index b6b1dc4..595a9be 100644
--- a/include/scsi/scsi_device.h
+++ b/include/scsi/scsi_device.h
@@ -413,6 +413,7 @@ static inline int scsi_execute_req(struct scsi_device *sdev,
 }
 extern void sdev_disable_disk_events(struct scsi_device *sdev);
 extern void sdev_enable_disk_events(struct scsi_device *sdev);
+extern int scsi_vpd_lun_id(struct scsi_device *, char *, size_t);
 
 #ifdef CONFIG_PM
 extern int scsi_autopm_get_device(struct scsi_device *);
-- 
1.8.5.6

[PATCH 17/23] scsi_dh_alua: use unique device id

[Hannes Reinecke]

Use scsi_vpd_lun_id() to assign a unique device identification
to the alua port group structure.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 70 +++++++++++++++++++++++++++---
 1 file changed, 65 insertions(+), 5 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index dbe9ff2..c2b2100b 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -70,6 +70,8 @@ static DEFINE_SPINLOCK(port_group_lock);
 struct alua_port_group {
 	struct kref		kref;
 	struct list_head	node;
+	unsigned char		device_id_str[256];
+	int			device_id_size;
 	int			group_id;
 	int			tpgs;
 	int			state;
@@ -229,7 +231,9 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 {
 	unsigned char *d;
 	int group_id = -1;
-	struct alua_port_group *pg = NULL;
+	char device_id_str[256];
+	int device_id_size;
+	struct alua_port_group *tmp_pg, *pg = NULL;
 
 	if (!sdev->vpd_pg83)
 		return SCSI_DH_DEV_UNSUPP;
@@ -266,9 +270,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		h->tpgs = TPGS_MODE_NONE;
 		return SCSI_DH_DEV_UNSUPP;
 	}
+	device_id_size = scsi_vpd_lun_id(sdev, device_id_str, 256);
+	if (device_id_size <= 0) {
+		/*
+		 * Internal error: TPGS supported by no
+		 * device identifcation found.
+		 * Disable ALUA support.
+		 */
+		sdev_printk(KERN_INFO, sdev,
+			    "%s: No device descriptors found\n",
+			    ALUA_DH_NAME);
+		h->tpgs = TPGS_MODE_NONE;
+		return SCSI_DH_DEV_UNSUPP;
+	}
 	sdev_printk(KERN_INFO, sdev,
-		    "%s: port group %02x rel port %02x\n",
-		    ALUA_DH_NAME, group_id, h->rel_port);
+		    "%s: device %s port group %02x "
+		    "rel port %02x\n", ALUA_DH_NAME,
+		    device_id_str, group_id, h->rel_port);
+	spin_lock(&port_group_lock);
+	list_for_each_entry(tmp_pg, &port_group_list, node) {
+		if (tmp_pg->group_id != group_id)
+			continue;
+		if (tmp_pg->device_id_size != device_id_size)
+			continue;
+		if (strncmp(tmp_pg->device_id_str, device_id_str,
+			    device_id_size))
+			continue;
+		h->pg = tmp_pg;
+		kref_get(&tmp_pg->kref);
+		break;
+	}
+	spin_unlock(&port_group_lock);
+	if (h->pg)
+		return SCSI_DH_OK;
 
 	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
 	if (!pg) {
@@ -278,13 +312,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		/* Temporary failure, bypass */
 		return SCSI_DH_DEV_TEMP_BUSY;
 	}
+	if (device_id_size)
+		strncpy(pg->device_id_str, device_id_str, 256);
+	else
+		pg->device_id_str[0] = '\0';
+
+	pg->device_id_size = device_id_size;
 	pg->group_id = group_id;
 	pg->tpgs = h->tpgs;
 	pg->state = TPGS_STATE_OPTIMIZED;
 	kref_init(&pg->kref);
 	spin_lock(&port_group_lock);
-	list_add(&pg->node, &port_group_list);
-	h->pg = pg;
+	/*
+	 * Re-check list again to catch
+	 * concurrent updates
+	 */
+	list_for_each_entry(tmp_pg, &port_group_list, node) {
+		if (tmp_pg->group_id != pg->group_id)
+			continue;
+		if (tmp_pg->device_id_size != pg->device_id_size)
+			continue;
+		if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
+			    device_id_size))
+			continue;
+		h->pg = tmp_pg;
+		kref_get(&tmp_pg->kref);
+		kfree(pg);
+		pg = NULL;
+		break;
+	}
+	if (pg) {
+		list_add(&pg->node, &port_group_list);
+		h->pg = pg;
+	}
 	spin_unlock(&port_group_lock);
 
 	return SCSI_DH_OK;
-- 
1.8.5.6

[PATCH 18/23] revert "scsi_dh_alua: ALUA hander attach should succeed while TPG is transitioning"

[Hannes Reinecke]

This reverts commit a8e5a2d593cbfccf530c3382c2c328d2edaa7b66

Obsoleted by the next patch.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 29 +++++++++++------------------
 1 file changed, 11 insertions(+), 18 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index c2b2100b..b52db8b 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -418,13 +418,12 @@ static int alua_check_sense(struct scsi_device *sdev,
 /*
  * alua_rtpg - Evaluate REPORT TARGET GROUP STATES
  * @sdev: the device to be evaluated.
- * @wait_for_transition: if nonzero, wait ALUA_FAILOVER_TIMEOUT seconds for device to exit transitioning state
  *
  * Evaluate the Target Port Group State.
  * Returns SCSI_DH_DEV_OFFLINED if the path is
  * found to be unusable.
  */
-static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
+static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 {
 	struct scsi_sense_hdr sense_hdr;
 	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
@@ -517,8 +516,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 	else
 		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
 
-	if (wait_for_transition &&
-	    (orig_transition_tmo != pg->transition_tmo)) {
+	if (orig_transition_tmo != pg->transition_tmo) {
 		sdev_printk(KERN_INFO, sdev,
 			    "%s: transition timeout set to %d seconds\n",
 			    ALUA_DH_NAME, pg->transition_tmo);
@@ -556,19 +554,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
 
 	switch (pg->state) {
 	case TPGS_STATE_TRANSITIONING:
-		if (wait_for_transition) {
-			if (time_before(jiffies, expiry)) {
-				/* State transition, retry */
-				interval += 2000;
-				msleep(interval);
-				goto retry;
-			}
-			err = SCSI_DH_RETRY;
-		} else {
-			err = SCSI_DH_OK;
+		if (time_before(jiffies, expiry)) {
+			/* State transition, retry */
+			interval += 2000;
+			msleep(interval);
+			goto retry;
 		}
-
 		/* Transitioning time exceeded, set port to standby */
+		err = SCSI_DH_RETRY;
 		pg->state = TPGS_STATE_STANDBY;
 		break;
 	case TPGS_STATE_OFFLINE:
@@ -665,7 +658,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
 		goto out;
 
 	kref_get(&h->pg->kref);
-	err = alua_rtpg(sdev, h->pg, 0);
+	err = alua_rtpg(sdev, h->pg);
 	kref_put(&h->pg->kref, release_port_group);
 out:
 	return err;
@@ -739,14 +732,14 @@ static int alua_activate(struct scsi_device *sdev,
 	if (optimize_stpg)
 		h->pg->flags |= ALUA_OPTIMIZE_STPG;
 
-	err = alua_rtpg(sdev, h->pg, 1);
+	err = alua_rtpg(sdev, h->pg);
 	if (err != SCSI_DH_OK) {
 		kref_put(&h->pg->kref, release_port_group);
 		goto out;
 	}
 	err = alua_stpg(sdev, h->pg);
 	if (err == SCSI_DH_RETRY)
-		err = alua_rtpg(sdev, h->pg, 1);
+		err = alua_rtpg(sdev, h->pg);
 	kref_put(&h->pg->kref, release_port_group);
 out:
 	if (fn)
-- 
1.8.5.6

[PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Hannes Reinecke]

The current ALUA device_handler has two drawbacks:
- We're sending a 'SET TARGET PORT GROUP' command to every LUN,
  disregarding the fact that several LUNs might be in a port group
  and will be automatically switched whenever _any_ LUN within
  that port group receives the command.
- Whenever a LUN is in 'transitioning' mode we cannot block I/O
  to that LUN, instead the controller has to abort the command.
  This leads to increased traffic across the wire and heavy load
  on the controller during switchover.

With this patch the RTPG handling is moved to a per-portgroup
workqueue. This reduces the number of 'REPORT TARGET PORT GROUP'
and 'SET TARGET PORT GROUPS' sent to the controller as we're sending
them now per port group, and not per device as previously.
It also allows us to block I/O to any LUN / port group found to be
in 'transitioning' ALUA mode, as the workqueue item will be requeued
until the controller moves out of transitioning.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 414 ++++++++++++++++++++++++-----
 1 file changed, 349 insertions(+), 65 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index b52db8b..85fd597 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -22,6 +22,8 @@
 #include <linux/slab.h>
 #include <linux/delay.h>
 #include <linux/module.h>
+#include <linux/workqueue.h>
+#include <linux/rcupdate.h>
 #include <asm/unaligned.h>
 #include <scsi/scsi.h>
 #include <scsi/scsi_dbg.h>
@@ -59,10 +61,15 @@
 #define ALUA_RTPG_SIZE			128
 #define ALUA_FAILOVER_TIMEOUT		60
 #define ALUA_FAILOVER_RETRIES		5
+#define ALUA_RTPG_DELAY_MSECS		5
 
 /* device handler flags */
-#define ALUA_OPTIMIZE_STPG		1
-#define ALUA_RTPG_EXT_HDR_UNSUPP	2
+#define ALUA_OPTIMIZE_STPG		0x01
+#define ALUA_RTPG_EXT_HDR_UNSUPP	0x02
+/* State machine flags */
+#define ALUA_PG_RUN_RTPG		0x10
+#define ALUA_PG_RUN_STPG		0x20
+
 
 static LIST_HEAD(port_group_list);
 static DEFINE_SPINLOCK(port_group_lock);
@@ -78,13 +85,28 @@ struct alua_port_group {
 	int			pref;
 	unsigned		flags; /* used for optimizing STPG */
 	unsigned char		transition_tmo;
+	unsigned long		expiry;
+	unsigned long		interval;
+	struct workqueue_struct *work_q;
+	struct delayed_work	rtpg_work;
+	struct delayed_work	stpg_work;
+	struct delayed_work	qdata_work;
+	spinlock_t		rtpg_lock;
+	struct list_head	rtpg_list;
+	struct scsi_device	*rtpg_sdev;
 };
 
 struct alua_dh_data {
 	struct alua_port_group	*pg;
+	spinlock_t		pg_lock;
 	int			rel_port;
 	int			tpgs;
-	struct scsi_device	*sdev;
+	int			init_error;
+	struct mutex		init_mutex;
+};
+
+struct alua_queue_data {
+	struct list_head	entry;
 	activate_complete	callback_fn;
 	void			*callback_data;
 };
@@ -93,6 +115,10 @@ struct alua_dh_data {
 #define ALUA_POLICY_SWITCH_ALL		1
 
 static char print_alua_state(int);
+static void alua_rtpg_work(struct work_struct *work);
+static void alua_stpg_work(struct work_struct *work);
+static void alua_qdata_work(struct work_struct *work);
+static void alua_check(struct scsi_device *sdev);
 
 static void release_port_group(struct kref *kref)
 {
@@ -103,6 +129,9 @@ static void release_port_group(struct kref *kref)
 	spin_lock(&port_group_lock);
 	list_del(&pg->node);
 	spin_unlock(&port_group_lock);
+	WARN_ON(pg->rtpg_sdev);
+	if (pg->work_q)
+		destroy_workqueue(pg->work_q);
 	kfree(pg);
 }
 
@@ -296,13 +325,17 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		if (strncmp(tmp_pg->device_id_str, device_id_str,
 			    device_id_size))
 			continue;
-		h->pg = tmp_pg;
 		kref_get(&tmp_pg->kref);
+		spin_lock(&h->pg_lock);
+		rcu_assign_pointer(h->pg, tmp_pg);
+		spin_unlock(&h->pg_lock);
 		break;
 	}
 	spin_unlock(&port_group_lock);
-	if (h->pg)
+	if (h->pg) {
+		synchronize_rcu();
 		return SCSI_DH_OK;
+	}
 
 	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
 	if (!pg) {
@@ -322,6 +355,19 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 	pg->tpgs = h->tpgs;
 	pg->state = TPGS_STATE_OPTIMIZED;
 	kref_init(&pg->kref);
+	INIT_DELAYED_WORK(&pg->rtpg_work, alua_rtpg_work);
+	INIT_DELAYED_WORK(&pg->stpg_work, alua_stpg_work);
+	INIT_DELAYED_WORK(&pg->qdata_work, alua_qdata_work);
+	INIT_LIST_HEAD(&pg->rtpg_list);
+	INIT_LIST_HEAD(&pg->node);
+	spin_lock_init(&pg->rtpg_lock);
+	pg->work_q = alloc_ordered_workqueue("alua_wp_%s_%d", WQ_MEM_RECLAIM,
+					     pg->device_id_str, pg->group_id);
+	if (!pg->work_q) {
+		kref_put(&pg->kref, release_port_group);
+		/* Temporary failure, bypass */
+		return SCSI_DH_DEV_TEMP_BUSY;
+	}
 	spin_lock(&port_group_lock);
 	/*
 	 * Re-check list again to catch
@@ -335,15 +381,19 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
 		if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
 			    device_id_size))
 			continue;
-		h->pg = tmp_pg;
 		kref_get(&tmp_pg->kref);
+		spin_lock(&h->pg_lock);
+		rcu_assign_pointer(h->pg, tmp_pg);
+		spin_unlock(&h->pg_lock);
 		kfree(pg);
 		pg = NULL;
 		break;
 	}
 	if (pg) {
 		list_add(&pg->node, &port_group_list);
-		h->pg = pg;
+		spin_lock(&h->pg_lock);
+		rcu_assign_pointer(h->pg, pg);
+		spin_unlock(&h->pg_lock);
 	}
 	spin_unlock(&port_group_lock);
 
@@ -377,11 +427,14 @@ static int alua_check_sense(struct scsi_device *sdev,
 {
 	switch (sense_hdr->sense_key) {
 	case NOT_READY:
-		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0a)
+		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0a) {
 			/*
 			 * LUN Not Accessible - ALUA state transition
+			 * Kickoff worker to update internal state.
 			 */
+			alua_check(sdev);
 			return ADD_TO_MLQUEUE;
+		}
 		break;
 	case UNIT_ATTENTION:
 		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
@@ -429,14 +482,15 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
 	unsigned char *ucp, *buff;
 	unsigned err, retval;
-	unsigned long expiry, interval = 0;
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 
-	if (!pg->transition_tmo)
-		expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
-	else
-		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
+	if (!pg->expiry) {
+		if (!pg->transition_tmo)
+			pg->expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
+		else
+			pg->expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
+	}
 
 	buff = kzalloc(bufflen, GFP_KERNEL);
 	if (!buff)
@@ -455,6 +509,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			else
 				err = SCSI_DH_IO;
 			kfree(buff);
+			pg->expiry = 0;
 			return err;
 		}
 
@@ -481,16 +536,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			err = SCSI_DH_RETRY;
 		else if (sense_hdr.sense_key == UNIT_ATTENTION)
 			err = SCSI_DH_RETRY;
-		if (err == SCSI_DH_RETRY && time_before(jiffies, expiry)) {
+		if (err == SCSI_DH_RETRY &&
+		    pg->expiry != 0 && time_before(jiffies, pg->expiry)) {
 			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
 				    ALUA_DH_NAME);
 			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
-			goto retry;
+			return err;
 		}
 		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
 			    ALUA_DH_NAME);
 		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
 		kfree(buff);
+		pg->expiry = 0;
 		return SCSI_DH_IO;
 	}
 
@@ -505,6 +562,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			sdev_printk(KERN_WARNING, sdev,
 				    "%s: kmalloc buffer failed\n",__func__);
 			/* Temporary failure, bypass */
+			pg->expiry = 0;
 			return SCSI_DH_DEV_TEMP_BUSY;
 		}
 		goto retry;
@@ -520,7 +578,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 		sdev_printk(KERN_INFO, sdev,
 			    "%s: transition timeout set to %d seconds\n",
 			    ALUA_DH_NAME, pg->transition_tmo);
-		expiry = jiffies + pg->transition_tmo * HZ;
+		pg->expiry = jiffies + pg->transition_tmo * HZ;
 	}
 
 	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
@@ -554,23 +612,26 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 
 	switch (pg->state) {
 	case TPGS_STATE_TRANSITIONING:
-		if (time_before(jiffies, expiry)) {
+		if (time_before(jiffies, pg->expiry)) {
 			/* State transition, retry */
-			interval += 2000;
-			msleep(interval);
-			goto retry;
+			pg->interval = 2;
+			err = SCSI_DH_RETRY;
+		} else {
+			/* Transitioning time exceeded, set port to standby */
+			err = SCSI_DH_IO;
+			pg->state = TPGS_STATE_STANDBY;
+			pg->expiry = 0;
 		}
-		/* Transitioning time exceeded, set port to standby */
-		err = SCSI_DH_RETRY;
-		pg->state = TPGS_STATE_STANDBY;
 		break;
 	case TPGS_STATE_OFFLINE:
 		/* Path unusable */
 		err = SCSI_DH_DEV_OFFLINED;
+		pg->expiry = 0;
 		break;
 	default:
 		/* Useable path if active */
 		err = SCSI_DH_OK;
+		pg->expiry = 0;
 		break;
 	}
 	kfree(buff);
@@ -590,8 +651,8 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	struct scsi_sense_hdr sense_hdr;
 
 	if (!(pg->tpgs & TPGS_MODE_EXPLICIT)) {
-		/* Only implicit ALUA supported, retry */
-		return SCSI_DH_RETRY;
+		/* Only implicit ALUA supported */
+		return SCSI_DH_OK;
 	}
 	switch (pg->state) {
 	case TPGS_STATE_OPTIMIZED:
@@ -617,8 +678,6 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
 		return SCSI_DH_NOSYS;
 		break;
 	}
-	/* Set state to transitioning */
-	pg->state = TPGS_STATE_TRANSITIONING;
 	retval = submit_stpg(sdev, pg->group_id, &sense_hdr);
 
 	if (retval) {
@@ -638,6 +697,150 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	return SCSI_DH_RETRY;
 }
 
+static void alua_rtpg_work(struct work_struct *work)
+{
+	struct alua_port_group *pg =
+		container_of(work, struct alua_port_group, rtpg_work.work);
+	struct scsi_device *sdev;
+	int err = SCSI_DH_OK;
+	unsigned long flags;
+
+	spin_lock_irqsave(&pg->rtpg_lock, flags);
+	sdev = pg->rtpg_sdev;
+	if (!sdev) {
+		WARN_ON(pg->flags & ALUA_PG_RUN_RTPG ||
+			pg->flags & ALUA_PG_RUN_STPG);
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		return;
+	}
+	if (pg->flags & ALUA_PG_RUN_RTPG) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		err = alua_rtpg(sdev, pg);
+		if (err == SCSI_DH_RETRY) {
+			queue_delayed_work(pg->work_q, &pg->rtpg_work,
+					   pg->interval * HZ);
+			return;
+		}
+		spin_lock_irqsave(&pg->rtpg_lock, flags);
+		pg->flags &= ~ALUA_PG_RUN_RTPG;
+		if (err != SCSI_DH_OK)
+			pg->flags &= ~ALUA_PG_RUN_STPG;
+	}
+	if (pg->flags & ALUA_PG_RUN_STPG) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		queue_delayed_work(pg->work_q, &pg->stpg_work, 0);
+		return;
+	}
+	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+	queue_delayed_work(pg->work_q, &pg->qdata_work, 0);
+}
+
+static void alua_stpg_work(struct work_struct *work)
+{
+	struct alua_port_group *pg =
+		container_of(work, struct alua_port_group, stpg_work.work);
+	struct scsi_device *sdev;
+	int err = SCSI_DH_OK;
+	unsigned long flags;
+
+	spin_lock_irqsave(&pg->rtpg_lock, flags);
+	sdev = pg->rtpg_sdev;
+	if (!sdev) {
+		WARN_ON(pg->flags & ALUA_PG_RUN_STPG);
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		return;
+	}
+
+	if (pg->flags & ALUA_PG_RUN_STPG) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		err = alua_stpg(sdev, pg);
+		spin_lock_irqsave(&pg->rtpg_lock, flags);
+		pg->flags &= ~ALUA_PG_RUN_STPG;
+		if (err == SCSI_DH_RETRY) {
+			pg->flags |= ALUA_PG_RUN_RTPG;
+			pg->interval = 0;
+		}
+	}
+	if (pg->flags & ALUA_PG_RUN_RTPG) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		queue_delayed_work(pg->work_q, &pg->rtpg_work,
+				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
+		return;
+	}
+
+	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+	queue_delayed_work(pg->work_q, &pg->qdata_work, 0);
+}
+
+static void alua_qdata_work(struct work_struct *work)
+{
+	struct alua_port_group *pg =
+		container_of(work, struct alua_port_group, qdata_work.work);
+	struct scsi_device *sdev;
+	LIST_HEAD(qdata_list);
+	int err = SCSI_DH_OK;
+	struct alua_queue_data *qdata, *tmp;
+	unsigned long flags;
+
+	spin_lock_irqsave(&pg->rtpg_lock, flags);
+	sdev = pg->rtpg_sdev;
+	if (WARN_ON(!sdev)) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		return;
+	}
+	if (pg->flags & ALUA_PG_RUN_RTPG) {
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		queue_delayed_work(pg->work_q, &pg->rtpg_work,
+				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
+		return;
+	}
+
+	list_splice_init(&pg->rtpg_list, &qdata_list);
+	pg->rtpg_sdev = NULL;
+	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+
+	list_for_each_entry_safe(qdata, tmp, &qdata_list, entry) {
+		list_del(&qdata->entry);
+		if (qdata->callback_fn)
+			qdata->callback_fn(qdata->callback_data, err);
+		kfree(qdata);
+	}
+	kref_put(&pg->kref, release_port_group);
+	scsi_device_put(sdev);
+}
+
+static void alua_rtpg_queue(struct alua_port_group *pg,
+			    struct scsi_device *sdev,
+			    struct alua_queue_data *qdata)
+{
+	int start_queue = 0;
+	unsigned long flags;
+
+	if (!pg)
+		return;
+
+	kref_get(&pg->kref);
+	spin_lock_irqsave(&pg->rtpg_lock, flags);
+	if (qdata) {
+		list_add_tail(&qdata->entry, &pg->rtpg_list);
+		pg->flags |= ALUA_PG_RUN_STPG;
+	}
+	if (pg->rtpg_sdev == NULL) {
+		pg->interval = 0;
+		pg->flags |= ALUA_PG_RUN_RTPG;
+		kref_get(&pg->kref);
+		pg->rtpg_sdev = sdev;
+		scsi_device_get(sdev);
+		start_queue = 1;
+	}
+	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+
+	if (start_queue)
+		queue_delayed_work(pg->work_q, &pg->rtpg_work,
+				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
+	kref_put(&pg->kref, release_port_group);
+}
+
 /*
  * alua_initialize - Initialize ALUA state
  * @sdev: the device to be initialized
@@ -647,22 +850,35 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
  */
 static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
 {
-	int err;
-
-	err = alua_check_tpgs(sdev, h);
-	if (err != SCSI_DH_OK)
-		goto out;
-
-	err = alua_check_vpd(sdev, h);
-	if (err != SCSI_DH_OK || !h->pg)
-		goto out;
-
-	kref_get(&h->pg->kref);
-	err = alua_rtpg(sdev, h->pg);
-	kref_put(&h->pg->kref, release_port_group);
-out:
-	return err;
+	struct alua_port_group *pg = NULL;
+	int error;
+
+	mutex_lock(&h->init_mutex);
+	error = alua_check_tpgs(sdev, h);
+	if (error == SCSI_DH_OK) {
+		error = alua_check_vpd(sdev, h);
+		rcu_read_lock();
+		pg = rcu_dereference(h->pg);
+		if (!pg) {
+			rcu_read_unlock();
+			h->tpgs = TPGS_MODE_NONE;
+			error = SCSI_DH_DEV_UNSUPP;
+		} else {
+			WARN_ON(error != SCSI_DH_OK);
+			kref_get(&pg->kref);
+			rcu_read_unlock();
+		}
+	}
+	h->init_error = error;
+	mutex_unlock(&h->init_mutex);
+	if (pg) {
+		pg->expiry = 0;
+		alua_rtpg_queue(pg, sdev, NULL);
+		kref_put(&pg->kref, release_port_group);
+	}
+	return error;
 }
+
 /*
  * alua_set_params - set/unset the optimize flag
  * @sdev: device on the path to be activated
@@ -679,6 +895,10 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
 	unsigned int optimize = 0, argc;
 	const char *p = params;
 	int result = SCSI_DH_OK;
+	unsigned long flags;
+
+	if (!h)
+		return -ENXIO;
 
 	if ((sscanf(params, "%u", &argc) != 1) || (argc != 1))
 		return -EINVAL;
@@ -694,11 +914,12 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
 		rcu_read_unlock();
 		return -ENXIO;
 	}
-
+	spin_lock_irqsave(&pg->rtpg_lock, flags);
 	if (optimize)
 		pg->flags |= ALUA_OPTIMIZE_STPG;
 	else
 		pg->flags |= ~ALUA_OPTIMIZE_STPG;
+	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
 	rcu_read_unlock();
 
 	return result;
@@ -723,24 +944,46 @@ static int alua_activate(struct scsi_device *sdev,
 {
 	struct alua_dh_data *h = sdev->handler_data;
 	int err = SCSI_DH_OK;
+	struct alua_queue_data *qdata;
+	struct alua_port_group *pg;
 
-	if (!h->pg)
+	if (!h) {
+		err = SCSI_DH_NOSYS;
 		goto out;
+	}
 
-	kref_get(&h->pg->kref);
-
-	if (optimize_stpg)
-		h->pg->flags |= ALUA_OPTIMIZE_STPG;
+	qdata = kzalloc(sizeof(*qdata), GFP_KERNEL);
+	if (!qdata) {
+		err = SCSI_DH_RES_TEMP_UNAVAIL;
+		goto out;
+	}
+	qdata->callback_fn = fn;
+	qdata->callback_data = data;
 
-	err = alua_rtpg(sdev, h->pg);
-	if (err != SCSI_DH_OK) {
-		kref_put(&h->pg->kref, release_port_group);
+	mutex_lock(&h->init_mutex);
+	rcu_read_lock();
+	pg = rcu_dereference(h->pg);
+	if (!pg) {
+		rcu_read_unlock();
+		kfree(qdata);
+		err = h->init_error;
+		mutex_unlock(&h->init_mutex);
 		goto out;
 	}
-	err = alua_stpg(sdev, h->pg);
-	if (err == SCSI_DH_RETRY)
-		err = alua_rtpg(sdev, h->pg);
-	kref_put(&h->pg->kref, release_port_group);
+	mutex_unlock(&h->init_mutex);
+	fn = NULL;
+	kref_get(&pg->kref);
+	rcu_read_unlock();
+
+	if (optimize_stpg) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&pg->rtpg_lock, flags);
+		pg->flags |= ALUA_OPTIMIZE_STPG;
+		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+	}
+	alua_rtpg_queue(pg, sdev, qdata);
+	kref_put(&pg->kref, release_port_group);
 out:
 	if (fn)
 		fn(data, err);
@@ -748,6 +991,30 @@ out:
 }
 
 /*
+ * alua_check - check path status
+ * @sdev: device on the path to be checked
+ *
+ * Check the device status
+ */
+static void alua_check(struct scsi_device *sdev)
+{
+	struct alua_dh_data *h = sdev->handler_data;
+	struct alua_port_group *pg;
+
+	rcu_read_lock();
+	pg = rcu_dereference(h->pg);
+	if (!pg) {
+		rcu_read_unlock();
+		return;
+	}
+	kref_get(&pg->kref);
+	rcu_read_unlock();
+	alua_rtpg_queue(pg, sdev, NULL);
+	kref_put(&pg->kref, release_port_group);
+	rcu_read_unlock();
+}
+
+/*
  * alua_prep_fn - request callback
  *
  * Fail I/O to all paths not in state
@@ -756,14 +1023,22 @@ out:
 static int alua_prep_fn(struct scsi_device *sdev, struct request *req)
 {
 	struct alua_dh_data *h = sdev->handler_data;
-	int state;
+	struct alua_port_group *pg;
+	int state = TPGS_STATE_OPTIMIZED;
 	int ret = BLKPREP_OK;
 
-	if (!h->pg)
+	if (!h)
 		return ret;
-	kref_get(&h->pg->kref);
-	state = h->pg->state;
-	kref_put(&h->pg->kref, release_port_group);
+
+	rcu_read_lock();
+	pg = rcu_dereference(h->pg);
+	if (pg) {
+		state = pg->state;
+		/* Defer I/O while rtpg_work is active */
+		if (pg->rtpg_sdev)
+			state = TPGS_STATE_TRANSITIONING;
+	}
+	rcu_read_unlock();
 	if (state == TPGS_STATE_TRANSITIONING)
 		ret = BLKPREP_DEFER;
 	else if (state != TPGS_STATE_OPTIMIZED &&
@@ -788,11 +1063,13 @@ static int alua_bus_attach(struct scsi_device *sdev)
 	h = kzalloc(sizeof(*h) , GFP_KERNEL);
 	if (!h)
 		return -ENOMEM;
+	spin_lock_init(&h->pg_lock);
 	h->tpgs = TPGS_MODE_UNINITIALIZED;
-	h->pg = NULL;
+	rcu_assign_pointer(h->pg, NULL);
 	h->rel_port = -1;
-	h->sdev = sdev;
+	h->init_error = SCSI_DH_OK;
 
+	mutex_init(&h->init_mutex);
 	err = alua_initialize(sdev, h);
 	if (err != SCSI_DH_OK && err != SCSI_DH_DEV_OFFLINED)
 		goto failed;
@@ -811,10 +1088,17 @@ failed:
 static void alua_bus_detach(struct scsi_device *sdev)
 {
 	struct alua_dh_data *h = sdev->handler_data;
+	struct alua_port_group *pg;
 
-	if (h->pg) {
-		kref_put(&h->pg->kref, release_port_group);
-		h->pg = NULL;
+	spin_lock(&h->pg_lock);
+	pg = h->pg;
+	rcu_assign_pointer(h->pg, NULL);
+	spin_unlock(&h->pg_lock);
+	synchronize_rcu();
+	if (pg) {
+		if (pg->rtpg_sdev)
+			flush_workqueue(pg->work_q);
+		kref_put(&pg->kref, release_port_group);
 	}
 	sdev->handler_data = NULL;
 	kfree(h);
-- 
1.8.5.6

[PATCH 20/23] scsi_dh_alua: Recheck state on unit attention

[Hannes Reinecke]

When we receive a unit attention code of 'ALUA state changed'
we should recheck the state, as it might be due to an implicit
ALUA state transition.
At the same time a workqueue item might already be queued, which
should be started immediately to avoid any delays.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 48 +++++++++++++++---------------
 1 file changed, 24 insertions(+), 24 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 85fd597..8717fd3 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -118,7 +118,7 @@ static char print_alua_state(int);
 static void alua_rtpg_work(struct work_struct *work);
 static void alua_stpg_work(struct work_struct *work);
 static void alua_qdata_work(struct work_struct *work);
-static void alua_check(struct scsi_device *sdev);
+static void alua_check(struct scsi_device *sdev, bool force);
 
 static void release_port_group(struct kref *kref)
 {
@@ -423,7 +423,7 @@ static char print_alua_state(int state)
 }
 
 static int alua_check_sense(struct scsi_device *sdev,
-			    struct scsi_sense_hdr *sense_hdr)
+			     struct scsi_sense_hdr *sense_hdr)
 {
 	switch (sense_hdr->sense_key) {
 	case NOT_READY:
@@ -432,36 +432,34 @@ static int alua_check_sense(struct scsi_device *sdev,
 			 * LUN Not Accessible - ALUA state transition
 			 * Kickoff worker to update internal state.
 			 */
-			alua_check(sdev);
-			return ADD_TO_MLQUEUE;
+			alua_check(sdev, false);
+			return NEEDS_RETRY;
 		}
 		break;
 	case UNIT_ATTENTION:
-		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
-			/*
-			 * Power On, Reset, or Bus Device Reset, just retry.
-			 */
-			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x04)
-			/*
-			 * Device internal reset
-			 */
-			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
+		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) {
 			/*
-			 * Mode Parameter Changed
+			 * Power On, Reset, or Bus Device Reset.
+			 * Might have obscured a state transition,
+			 * so schedule a recheck.
 			 */
+			alua_check(sdev, true);
 			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
+		}
+		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06) {
 			/*
 			 * ALUA state changed
 			 */
+			alua_check(sdev, true);
 			return ADD_TO_MLQUEUE;
-		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07)
+		}
+		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07) {
 			/*
 			 * Implicit ALUA state transition failed
 			 */
+			alua_check(sdev, true);
 			return ADD_TO_MLQUEUE;
+		}
 		break;
 	}
 
@@ -811,7 +809,7 @@ static void alua_qdata_work(struct work_struct *work)
 
 static void alua_rtpg_queue(struct alua_port_group *pg,
 			    struct scsi_device *sdev,
-			    struct alua_queue_data *qdata)
+			    struct alua_queue_data *qdata, bool force)
 {
 	int start_queue = 0;
 	unsigned long flags;
@@ -832,7 +830,9 @@ static void alua_rtpg_queue(struct alua_port_group *pg,
 		pg->rtpg_sdev = sdev;
 		scsi_device_get(sdev);
 		start_queue = 1;
-	}
+	} else if (!(pg->flags & ALUA_PG_RUN_RTPG) && force)
+		start_queue = 1;
+
 	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
 
 	if (start_queue)
@@ -873,7 +873,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
 	mutex_unlock(&h->init_mutex);
 	if (pg) {
 		pg->expiry = 0;
-		alua_rtpg_queue(pg, sdev, NULL);
+		alua_rtpg_queue(pg, sdev, NULL, true);
 		kref_put(&pg->kref, release_port_group);
 	}
 	return error;
@@ -982,7 +982,7 @@ static int alua_activate(struct scsi_device *sdev,
 		pg->flags |= ALUA_OPTIMIZE_STPG;
 		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
 	}
-	alua_rtpg_queue(pg, sdev, qdata);
+	alua_rtpg_queue(pg, sdev, qdata, true);
 	kref_put(&pg->kref, release_port_group);
 out:
 	if (fn)
@@ -996,7 +996,7 @@ out:
  *
  * Check the device status
  */
-static void alua_check(struct scsi_device *sdev)
+static void alua_check(struct scsi_device *sdev, bool force)
 {
 	struct alua_dh_data *h = sdev->handler_data;
 	struct alua_port_group *pg;
@@ -1009,7 +1009,7 @@ static void alua_check(struct scsi_device *sdev)
 	}
 	kref_get(&pg->kref);
 	rcu_read_unlock();
-	alua_rtpg_queue(pg, sdev, NULL);
+	alua_rtpg_queue(pg, sdev, NULL, force);
 	kref_put(&pg->kref, release_port_group);
 	rcu_read_unlock();
 }
-- 
1.8.5.6

[PATCH 21/23] scsi_dh_alua: update all port states

[Hannes Reinecke]

When we read in the target port group state we should be
updating all affected port groups, otherwise we risk
running out of sync.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 8717fd3..2fa985e 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -477,11 +477,13 @@ static int alua_check_sense(struct scsi_device *sdev,
 static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 {
 	struct scsi_sense_hdr sense_hdr;
+	struct alua_port_group *tmp_pg;
 	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
 	unsigned char *ucp, *buff;
 	unsigned err, retval;
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
+	unsigned long flags;
 
 	if (!pg->expiry) {
 		if (!pg->transition_tmo)
@@ -584,17 +586,28 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	else
 		tpg_desc_tbl_off = 4;
 
+	spin_lock_irqsave(&port_group_lock, flags);
 	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
 	     k < len;
 	     k += off, ucp += off) {
 
-		if (pg->group_id == get_unaligned_be16(&ucp[2])) {
-			pg->state = ucp[0] & 0x0f;
-			pg->pref = ucp[0] >> 7;
-			valid_states = ucp[1];
+		list_for_each_entry(tmp_pg, &port_group_list, node) {
+			u16 group_id = get_unaligned_be16(&ucp[2]);
+			if (tmp_pg->group_id != group_id)
+				continue;
+			if (tmp_pg->device_id_size != pg->device_id_size)
+				continue;
+			if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
+				    tmp_pg->device_id_size))
+				continue;
+			tmp_pg->state = ucp[0] & 0x0f;
+			tmp_pg->pref = ucp[0] >> 7;
+			if (tmp_pg == pg)
+				valid_states = ucp[1];
 		}
 		off = 8 + (ucp[7] * 4);
 	}
+	spin_unlock_irqrestore(&port_group_lock, flags);
 
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
-- 
1.8.5.6

[PATCH 22/23] scsi_dh_alua: Send TEST UNIT READY to poll for transitioning

[Hannes Reinecke]

Sending a 'REPORT TARGET PORT GROUP' command is a costly operation,
as the array has to gather information about all ports.
So instead of using RTPG to poll for a status update when a port
is in transitioning we should be sending a TEST UNIT READY, and
wait for the sense code to report success.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 33 ++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 2fa985e..3c6b365 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -467,6 +467,30 @@ static int alua_check_sense(struct scsi_device *sdev,
 }
 
 /*
+ * alua_tur - Send a TEST UNIT READY
+ * @sdev: device to which the TEST UNIT READY command should be send
+ *
+ * Send a TEST UNIT READY to @sdev to figure out the device state
+ * Returns SCSI_DH_RETRY if the sense code is NOT READY/ALUA TRANSITIONING,
+ * SCSI_DH_OK if no error occured, and SCSI_DH_IO otherwise.
+ */
+static int alua_tur(struct scsi_device *sdev)
+{
+	struct scsi_sense_hdr sense_hdr;
+	int retval;
+
+	retval = scsi_test_unit_ready(sdev, ALUA_FAILOVER_TIMEOUT * HZ,
+				      ALUA_FAILOVER_RETRIES, &sense_hdr);
+	if (sense_hdr.sense_key == NOT_READY &&
+	    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
+		return SCSI_DH_RETRY;
+	else if (retval)
+		return SCSI_DH_IO;
+	else
+		return SCSI_DH_OK;
+}
+
+/*
  * alua_rtpg - Evaluate REPORT TARGET GROUP STATES
  * @sdev: the device to be evaluated.
  *
@@ -725,7 +749,16 @@ static void alua_rtpg_work(struct work_struct *work)
 		return;
 	}
 	if (pg->flags & ALUA_PG_RUN_RTPG) {
+		int state = pg->state;
 		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
+		if (state == TPGS_STATE_TRANSITIONING) {
+			if (alua_tur(sdev) == SCSI_DH_RETRY) {
+				queue_delayed_work(pg->work_q, &pg->rtpg_work,
+						   pg->interval * HZ);
+				return;
+			}
+			/* Send RTPG on failure or if TUR indicates SUCCESS */
+		}
 		err = alua_rtpg(sdev, pg);
 		if (err == SCSI_DH_RETRY) {
 			queue_delayed_work(pg->work_q, &pg->rtpg_work,
-- 
1.8.5.6

[PATCH 23/23] scsi_dh_alua: Update version to 2.0

[Hannes Reinecke]

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/device_handler/scsi_dh_alua.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
index 3c6b365..5b15936 100644
--- a/drivers/scsi/device_handler/scsi_dh_alua.c
+++ b/drivers/scsi/device_handler/scsi_dh_alua.c
@@ -31,7 +31,7 @@
 #include <scsi/scsi_dh.h>
 
 #define ALUA_DH_NAME "alua"
-#define ALUA_DH_VER "1.3"
+#define ALUA_DH_VER "2.0"
 
 #define TPGS_STATE_OPTIMIZED		0x0
 #define TPGS_STATE_NONOPTIMIZED		0x1
-- 
1.8.5.6

Re: [PATCH 01/23] scsi_dh_alua: Disable ALUA handling for non-disk devices

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 04/23] scsi_dh_alua: use standard logging functions

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:02PM +0200, Hannes Reinecke wrote:
>  		}
>  
>  		err = alua_check_sense(sdev, &sense_hdr);
> -		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry))
> +		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
> +			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
> +				    ALUA_DH_NAME);
> +			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>  			goto retry;
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: rtpg sense code %02x/%02x/%02x\n",
> -			    ALUA_DH_NAME, sense_hdr.sense_key,
> -			    sense_hdr.asc, sense_hdr.ascq);
> -		err = SCSI_DH_IO;
> +		}
> +		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
> +			    ALUA_DH_NAME);
> +		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> +		return SCSI_DH_IO;
>  	}
> -	if (err != SCSI_DH_OK)
> -		return err;

I think you need to keep this if, given that submit_rtpg can return
others error than SCSI_DH_IO as well.

While you're at it you might remove the h->senselen check, and
kill the assignment of the scsi_normalize_sense bool return value
to the err variable similar to how you did in stpg_endio. 

Re: [PATCH 05/23] scsi_dh_alua: return standard SCSI return codes in submit_rtpg

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:03PM +0200, Hannes Reinecke wrote:
> Fixup submit_rtpg() to always return a standard SCSI return code.

Oh, this fixes the problems in the previous patch up.  Maybe just skip the
error handling changes in the previous patch and keep them purely in this
one?

> +			if (driver_byte(retval) == DRIVER_BUSY)
> +				err = SCSI_DH_DEV_TEMP_BUSY;
> +			else
> +				err = SCSI_DH_IO;
> +			return err;

This could be simplified to:

			if (driver_byte(retval) == DRIVER_BUSY)
				return SCSI_DH_DEV_TEMP_BUSY;
			return SCSI_DH_IO;

Re: [PATCH 06/23] scsi_dh_alua: fixup description of stpg_endio()

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 09/23] scsi_dh_alua: use unaligned access macros

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 10/23] scsi_dh_alua: Pass buffer as function argument

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:08PM +0200, Hannes Reinecke wrote:
> Pass in the buffer as a function argument for submit_rtpg().

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 11/23] scsi_dh_alua: Make stpg synchronous

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:09PM +0200, Hannes Reinecke wrote:
> We should be issuing STPG synchronously as we need to
> evaluate the return code on failure.

This looks right, but I think the patch needs to be split and needs
a way better changelog.

patch 1:
 - factor out the alua_stpg helper

patch 2:
 - async STPG
 - please explain that alua_activate will now block and why this is ok

patch 3:
 - add the new call to alua_rtpg and why it's added

Re: [PATCH 12/23] scsi_dh_alua: switch to scsi_execute_req_flags()

[Christoph Hellwig]

Looks good fine, but please add a comment explaining why the
REQ_QUIET and REQ_PREEMPT flags added by scsi_execture are fine,
or in fact desireable.

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Christoph Hellwig]

> +struct alua_dh_data {
> +	struct alua_port_group	*pg;
> +	int			rel_port;
> +	int			tpgs;

The tpgs file doesn't make much sense in the new alua_dh_data
structure.  Please return it explicitly from alua_check_tpgs and
assign it directly to the member in struct alua_port_group.

> +static void release_port_group(struct kref *kref)
> +{
> +	struct alua_port_group *pg;
> +
> +	pg = container_of(kref, struct alua_port_group, kref);
> +	printk(KERN_WARNING "alua: release port group %d\n", pg->group_id);

Why is this a warning?  I'd consider it nothing but debug noise.

> -	err = alua_rtpg(sdev, h, 0);
> -	if (err != SCSI_DH_OK)
> +	if (err != SCSI_DH_OK || !h->pg)
>  		goto out;
>  
> +	kref_get(&h->pg->kref);

What prevents this kref_get from racing with the last kref_put?

I think all the kref_get calls in this patch need to be
kref_get_unless_zero with proper error handling.

> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (!pg) {
> +		rcu_read_unlock();
> +		return -ENXIO;
> +	}

What is this rcu_read_lock supposed to protect against given
that struct alua_port_group isn't RCU freed?  I think this needs
to be another kref_get_unless_zero.

> +
>  	if (optimize)
> -		h->flags |= ALUA_OPTIMIZE_STPG;
> +		pg->flags |= ALUA_OPTIMIZE_STPG;
>  	else
> -		h->flags &= ~ALUA_OPTIMIZE_STPG;
> +		pg->flags |= ~ALUA_OPTIMIZE_STPG;
> +	rcu_read_unlock();

The read-modify-write of pg->flags will need some sort of locking.
Seems like most modifications of it are under pg->rtpg_lock, so
I'd suggest to enforce that as a strict rule.

> +	err = alua_rtpg(sdev, h->pg, 1);
> +	if (err != SCSI_DH_OK) {
> +		kref_put(&h->pg->kref, release_port_group);
> +		goto out;

		goto out_put_pg;


out_put_pg:
> +	kref_put(&h->pg->kref, release_port_group);
>  out:
>  	if (fn)
>  		fn(data, err);

Re: [PATCH 16/23] scsi: Add scsi_vpd_lun_id()

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:14PM +0200, Hannes Reinecke wrote:
> Add a function scsi_vpd_lun_id() to return a unique device
> identifcation based on the designation descriptors of
> VPD page 0x83.
> 
> As devices might implement several descriptors the order
> of preference is:
> - NAA IEE Registered Extended
> - EUI-64 based 16-byte
> - EUI-64 based 12-byte
> - NAA IEEE Registered
> - NAA IEEE Extended
> A SCSI name string descriptor is preferred to all of them
> if the identification is longer than 16 bytes.
> 
> The returned unique device identification will be formatted
> as a SCSI Name string to avoid clashes between different
> designator types.

Looks good in general, but I wonder if it might make sense to switch
to kasprintf and let the caller free the buffer?

Otherwise:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 17/23] scsi_dh_alua: use unique device id

[Christoph Hellwig]

> +		 * Internal error: TPGS supported by no
> +		 * device identifcation found.
> +		 * Disable ALUA support.

s/by/but/

Also I think the comment could fit into two lines if you try :)

> +	spin_lock(&port_group_lock);
> +	list_for_each_entry(tmp_pg, &port_group_list, node) {
> +		if (tmp_pg->group_id != group_id)
> +			continue;
> +		if (tmp_pg->device_id_size != device_id_size)
> +			continue;
> +		if (strncmp(tmp_pg->device_id_str, device_id_str,
> +			    device_id_size))
> +			continue;
> +		h->pg = tmp_pg;
> +		kref_get(&tmp_pg->kref);
> +		break;
> +	}
> +	spin_unlock(&port_group_lock);

This exact code appears twice in this patch, please factor it into
a helper.

> +	if (device_id_size)
> +		strncpy(pg->device_id_str, device_id_str, 256);
> +	else
> +		pg->device_id_str[0] = '\0';
> +

How could we end up with a zero device ID length?  Shouldn't
have error out earlier on that?

> +	 * Re-check list again to catch
> +	 * concurrent updates
> +	 */

Could fit onto one line..

Re: [PATCH 20/23] scsi_dh_alua: Recheck state on unit attention

[Christoph Hellwig]

> -			alua_check(sdev);
> -			return ADD_TO_MLQUEUE;
> +			alua_check(sdev, false);
> +			return NEEDS_RETRY;

What's the reason for the change in return value here?

Re: [PATCH 21/23] scsi_dh_alua: update all port states

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:19PM +0200, Hannes Reinecke wrote:
> When we read in the target port group state we should be
> updating all affected port groups, otherwise we risk
> running out of sync.

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 22/23] scsi_dh_alua: Send TEST UNIT READY to poll for transitioning

[Christoph Hellwig]

On Thu, Aug 27, 2015 at 02:41:20PM +0200, Hannes Reinecke wrote:
> Sending a 'REPORT TARGET PORT GROUP' command is a costly operation,
> as the array has to gather information about all ports.
> So instead of using RTPG to poll for a status update when a port
> is in transitioning we should be sending a TEST UNIT READY, and
> wait for the sense code to report success.

The extra state variabe doesn't really help you as it might be stale
just as quickly.

But otherwise looks fine:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 23/23] scsi_dh_alua: Update version to 2.0

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Christoph Hellwig]

Ok, coming back to this path as it's the start of somethign building
up over various patches:

> +	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
> +	if (!pg) {
> +		sdev_printk(KERN_WARNING, sdev,
> +			    "%s: kzalloc port group failed\n",
> +			    ALUA_DH_NAME);
> +		/* Temporary failure, bypass */
> +		return SCSI_DH_DEV_TEMP_BUSY;
> +	}
> +	pg->group_id = group_id;
> +	pg->buff = pg->inq;
> +	pg->bufflen = ALUA_INQUIRY_SIZE;
> +	pg->tpgs = h->tpgs;
> +	pg->state = TPGS_STATE_OPTIMIZED;
> +	kref_init(&pg->kref);
> +	spin_lock(&port_group_lock);
> +	list_add(&pg->node, &port_group_list);
> +	h->pg = pg;
> +	spin_unlock(&port_group_lock);
> +
> +	return SCSI_DH_OK;

All this code isn't really checking the VPD anymore.  Please keep
the existing alua_check_vpd as a low-level helper, and move this
new functionality into a separate alua_find_get_pg function that
calls the original alua_check_vpd.  Also please return the
pg from it so that we c

> @@ -596,13 +643,12 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  		goto out;
>  
>  	err = alua_check_vpd(sdev, h);
> -	if (err != SCSI_DH_OK)
> -		goto out;
> -
> -	err = alua_rtpg(sdev, h, 0);
> -	if (err != SCSI_DH_OK)
> +	if (err != SCSI_DH_OK || !h->pg)
>  		goto out;

How could we end up here without h->pg? Either way that should
move into a conditional together with the kref_get which should
become the not_zero variant if it is kept.

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Christoph Hellwig]

> +	unsigned long		expiry;
> +	unsigned long		interval;
> +	struct workqueue_struct *work_q;
> +	struct delayed_work	rtpg_work;
> +	struct delayed_work	stpg_work;
> +	struct delayed_work	qdata_work;
> +	spinlock_t		rtpg_lock;

This one also protects ->flag.  I'd just call it lock, and
introduce it at the time struct alua_port_group is introduced,
so that we can have coherent locking from the very beginning.

> +struct alua_queue_data {
> +	struct list_head	entry;
>  	activate_complete	callback_fn;
>  	void			*callback_data;

I've been looking over the active callback for a while, and I
think the model of it is fundamentally broken.  Yes, as long as
multipathing isn't in SCSI we'll need path change notifications
for dm-mpath, but I don't see how tying them into ->activate
makes any sense.

I guess for this series we're stuck with it, but in the long
run we should have a callback in the request_queue which the
consumer that has bd_claim()ed the device can set and get path
change notifications instead.

That being said after spending a lot of time reading this code I think
my orginal advice to use different work_structs for the different
kinds of events was wrong, and I'd like to apologize for making you
go down that direction.

STPG/RTPG is just too dependent to sensibly split them up, and the
qdata bit while broken can be shoe horned in for now.  So I'd suggest
to revert back to the original model with a single work_struct per
port group, and a global workqueue.

> -	if (h->pg)
> +	if (h->pg) {
> +		synchronize_rcu();
>  		return SCSI_DH_OK;
> +	}

What is this synchronize_rcu supposed to help with?

> -		h->pg = tmp_pg;
>  		kref_get(&tmp_pg->kref);
> +		spin_lock(&h->pg_lock);
> +		rcu_assign_pointer(h->pg, tmp_pg);
> +		spin_unlock(&h->pg_lock);

I think the assignment to h->ph should have been past the kref_get
from the very beginning, please fold this into the original patch.

> +	struct alua_port_group *pg = NULL;
> +	int error;
> +
> +	mutex_lock(&h->init_mutex);
> +	error = alua_check_tpgs(sdev, h);
> +	if (error == SCSI_DH_OK) {
> +		error = alua_check_vpd(sdev, h);
> +		rcu_read_lock();
> +		pg = rcu_dereference(h->pg);
> +		if (!pg) {
> +			rcu_read_unlock();
> +			h->tpgs = TPGS_MODE_NONE;
> +			error = SCSI_DH_DEV_UNSUPP;
> +		} else {
> +			WARN_ON(error != SCSI_DH_OK);
> +			kref_get(&pg->kref);
> +			rcu_read_unlock();
> +		}
> +	}

Eww. Please grab the reference to the pg inside the replacement for
alua_check_vpd, and ensure that we never return from that without a
valid h->pg.  Together with the previously suggested removal of h->tpgs,
and moving the call to alua_rtpg_queue into the alua_check_vpd replacement
that would keep alua_initialize nice and simple.

> @@ -679,6 +895,10 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
>  	unsigned int optimize = 0, argc;
>  	const char *p = params;
>  	int result = SCSI_DH_OK;
> +	unsigned long flags;
> +
> +	if (!h)
> +		return -ENXIO;

How could that happen?  Why does it beling into this patch?

> @@ -723,24 +944,46 @@ static int alua_activate(struct scsi_device *sdev,
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
>  	int err = SCSI_DH_OK;
> +	struct alua_queue_data *qdata;
> +	struct alua_port_group *pg;
>  
> -	if (!h->pg)
> +	if (!h) {
> +		err = SCSI_DH_NOSYS;
>  		goto out;
> +	}

Same here.

> +	qdata = kzalloc(sizeof(*qdata), GFP_KERNEL);
> +	if (!qdata) {
> +		err = SCSI_DH_RES_TEMP_UNAVAIL;
> +		goto out;
> +	}
> +	qdata->callback_fn = fn;
> +	qdata->callback_data = data;
>  
> +	mutex_lock(&h->init_mutex);
> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (!pg) {
> +		rcu_read_unlock();
> +		kfree(qdata);
> +		err = h->init_error;
> +		mutex_unlock(&h->init_mutex);
>  		goto out;
>  	}
> +	mutex_unlock(&h->init_mutex);
> +	fn = NULL;
> +	kref_get(&pg->kref);
> +	rcu_read_unlock();
> +
> +	if (optimize_stpg) {
> +		unsigned long flags;
> +
> +		spin_lock_irqsave(&pg->rtpg_lock, flags);
> +		pg->flags |= ALUA_OPTIMIZE_STPG;
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +	}

The optimize_stpg should have been moved towards the alua_port_group
allocation earlier in the series, please fold that in there.

> +	alua_rtpg_queue(pg, sdev, qdata);
> +	kref_put(&pg->kref, release_port_group);

Note that all alua_rtpg_queue callers already have a reference to the
PG, so I don't think we need to grab another one inside it or even
check for a NULL pg.

Re: [PATCH 04/23] scsi_dh_alua: use standard logging functions

[Hannes Reinecke]

On 09/01/2015 11:48 AM, Christoph Hellwig wrote:
> On Thu, Aug 27, 2015 at 02:41:02PM +0200, Hannes Reinecke wrote:
>>  		}
>>  
>>  		err = alua_check_sense(sdev, &sense_hdr);
>> -		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry))
>> +		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
>> +			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
>> +				    ALUA_DH_NAME);
>> +			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>>  			goto retry;
>> -		sdev_printk(KERN_INFO, sdev,
>> -			    "%s: rtpg sense code %02x/%02x/%02x\n",
>> -			    ALUA_DH_NAME, sense_hdr.sense_key,
>> -			    sense_hdr.asc, sense_hdr.ascq);
>> -		err = SCSI_DH_IO;
>> +		}
>> +		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
>> +			    ALUA_DH_NAME);
>> +		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>> +		return SCSI_DH_IO;
>>  	}
>> -	if (err != SCSI_DH_OK)
>> -		return err;
> 
> I think you need to keep this if, given that submit_rtpg can return
> others error than SCSI_DH_IO as well.
> 
> While you're at it you might remove the h->senselen check, and
> kill the assignment of the scsi_normalize_sense bool return value
> to the err variable similar to how you did in stpg_endio. 
> 
Okay, will be doing so.

Cheers,

Hannes

-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 16/23] scsi: Add scsi_vpd_lun_id()

[Hannes Reinecke]

On 09/01/2015 12:22 PM, Christoph Hellwig wrote:
> On Thu, Aug 27, 2015 at 02:41:14PM +0200, Hannes Reinecke wrote:
>> Add a function scsi_vpd_lun_id() to return a unique device
>> identifcation based on the designation descriptors of
>> VPD page 0x83.
>>
>> As devices might implement several descriptors the order
>> of preference is:
>> - NAA IEE Registered Extended
>> - EUI-64 based 16-byte
>> - EUI-64 based 12-byte
>> - NAA IEEE Registered
>> - NAA IEEE Extended
>> A SCSI name string descriptor is preferred to all of them
>> if the identification is longer than 16 bytes.
>>
>> The returned unique device identification will be formatted
>> as a SCSI Name string to avoid clashes between different
>> designator types.
> 
> Looks good in general, but I wonder if it might make sense to switch
> to kasprintf and let the caller free the buffer?
> 
I really am no friend of caller-needs-to-free-up-buffer thingies.
Allocations and deallocations should be done in the same context for
better tracking (and debugging), IMO.

> Otherwise:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> 
Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Hannes Reinecke]

On 09/01/2015 01:15 PM, Christoph Hellwig wrote:
>> +	unsigned long		expiry;
>> +	unsigned long		interval;
>> +	struct workqueue_struct *work_q;
>> +	struct delayed_work	rtpg_work;
>> +	struct delayed_work	stpg_work;
>> +	struct delayed_work	qdata_work;
>> +	spinlock_t		rtpg_lock;
> 
> This one also protects ->flag.  I'd just call it lock, and
> introduce it at the time struct alua_port_group is introduced,
> so that we can have coherent locking from the very beginning.
> 
Okay.

>> +struct alua_queue_data {
>> +	struct list_head	entry;
>>  	activate_complete	callback_fn;
>>  	void			*callback_data;
> 
> I've been looking over the active callback for a while, and I
> think the model of it is fundamentally broken.  Yes, as long as
> multipathing isn't in SCSI we'll need path change notifications
> for dm-mpath, but I don't see how tying them into ->activate
> makes any sense.
> 
> I guess for this series we're stuck with it, but in the long
> run we should have a callback in the request_queue which the
> consumer that has bd_claim()ed the device can set and get path
> change notifications instead.
> 
That is what I'm eventually planning to do.
My final goal is to move the multipath path monitoring stuff
into the kernel (via the block device polling mechanism), and sending
block events for path failure and re-establishment.

But that'll be subject to further patches :-)

> That being said after spending a lot of time reading this code I think
> my orginal advice to use different work_structs for the different
> kinds of events was wrong, and I'd like to apologize for making you
> go down that direction.
> 
> STPG/RTPG is just too dependent to sensibly split them up, and the
> qdata bit while broken can be shoe horned in for now.  So I'd suggest
> to revert back to the original model with a single work_struct per
> port group, and a global workqueue.
> 
Weelll ... there was a reason why I did that initially :-)
But anyway, no harm done. I still have the original series around.

>> -	if (h->pg)
>> +	if (h->pg) {
>> +		synchronize_rcu();
>>  		return SCSI_DH_OK;
>> +	}
> 
> What is this synchronize_rcu supposed to help with?
> 
>> -		h->pg = tmp_pg;
>>  		kref_get(&tmp_pg->kref);
>> +		spin_lock(&h->pg_lock);
>> +		rcu_assign_pointer(h->pg, tmp_pg);
>> +		spin_unlock(&h->pg_lock);
> 
> I think the assignment to h->ph should have been past the kref_get
> from the very beginning, please fold this into the original patch.
> 
Okay.

>> +	struct alua_port_group *pg = NULL;
>> +	int error;
>> +
>> +	mutex_lock(&h->init_mutex);
>> +	error = alua_check_tpgs(sdev, h);
>> +	if (error == SCSI_DH_OK) {
>> +		error = alua_check_vpd(sdev, h);
>> +		rcu_read_lock();
>> +		pg = rcu_dereference(h->pg);
>> +		if (!pg) {
>> +			rcu_read_unlock();
>> +			h->tpgs = TPGS_MODE_NONE;
>> +			error = SCSI_DH_DEV_UNSUPP;
>> +		} else {
>> +			WARN_ON(error != SCSI_DH_OK);
>> +			kref_get(&pg->kref);
>> +			rcu_read_unlock();
>> +		}
>> +	}
> 
> Eww. Please grab the reference to the pg inside the replacement for
> alua_check_vpd, and ensure that we never return from that without a
> valid h->pg.  Together with the previously suggested removal of h->tpgs,
> and moving the call to alua_rtpg_queue into the alua_check_vpd replacement
> that would keep alua_initialize nice and simple.
> 
You are right in that the interface for alua_check_tpgs isn't very
clear. Currently we need to check both the return code _and_ 'h->pg',
even though both are interrelated.
I guess it should rather be 'h->pg = alua_check_vpd()' or even making
alua_check_vpd() a void function, which would eliminate the need for the
separate return value.
I'll have to think about it.

>> @@ -679,6 +895,10 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
>>  	unsigned int optimize = 0, argc;
>>  	const char *p = params;
>>  	int result = SCSI_DH_OK;
>> +	unsigned long flags;
>> +
>> +	if (!h)
>> +		return -ENXIO;
> 
> How could that happen?  Why does it beling into this patch?
> 
Leftover from the original patch, which didn't have your scsi_dh
patchset. I'll remove it.

>> @@ -723,24 +944,46 @@ static int alua_activate(struct scsi_device *sdev,
>>  {
>>  	struct alua_dh_data *h = sdev->handler_data;
>>  	int err = SCSI_DH_OK;
>> +	struct alua_queue_data *qdata;
>> +	struct alua_port_group *pg;
>>  
>> -	if (!h->pg)
>> +	if (!h) {
>> +		err = SCSI_DH_NOSYS;
>>  		goto out;
>> +	}
> 
> Same here.
> 
>> +	qdata = kzalloc(sizeof(*qdata), GFP_KERNEL);
>> +	if (!qdata) {
>> +		err = SCSI_DH_RES_TEMP_UNAVAIL;
>> +		goto out;
>> +	}
>> +	qdata->callback_fn = fn;
>> +	qdata->callback_data = data;
>>  
>> +	mutex_lock(&h->init_mutex);
>> +	rcu_read_lock();
>> +	pg = rcu_dereference(h->pg);
>> +	if (!pg) {
>> +		rcu_read_unlock();
>> +		kfree(qdata);
>> +		err = h->init_error;
>> +		mutex_unlock(&h->init_mutex);
>>  		goto out;
>>  	}
>> +	mutex_unlock(&h->init_mutex);
>> +	fn = NULL;
>> +	kref_get(&pg->kref);
>> +	rcu_read_unlock();
>> +
>> +	if (optimize_stpg) {
>> +		unsigned long flags;
>> +
>> +		spin_lock_irqsave(&pg->rtpg_lock, flags);
>> +		pg->flags |= ALUA_OPTIMIZE_STPG;
>> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>> +	}
> 
> The optimize_stpg should have been moved towards the alua_port_group
> allocation earlier in the series, please fold that in there.
> 
Okay.

>> +	alua_rtpg_queue(pg, sdev, qdata);
>> +	kref_put(&pg->kref, release_port_group);
> 
> Note that all alua_rtpg_queue callers already have a reference to the
> PG, so I don't think we need to grab another one inside it or even
> check for a NULL pg.
> 
In general I try to follow the rule of getting an initial reference, and
another one whenever the port_group structure is put on the workqueue.
But I'll check here if the additional reference is warranted.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/23] scsi_dh_alua: Make stpg synchronous

[Hannes Reinecke]

On 09/01/2015 12:04 PM, Christoph Hellwig wrote:
> On Thu, Aug 27, 2015 at 02:41:09PM +0200, Hannes Reinecke wrote:
>> We should be issuing STPG synchronously as we need to
>> evaluate the return code on failure.
> 
> This looks right, but I think the patch needs to be split and needs
> a way better changelog.
> 
> patch 1:
>  - factor out the alua_stpg helper
> 
> patch 2:
>  - async STPG
>  - please explain that alua_activate will now block and why this is ok
> 
> patch 3:
>  - add the new call to alua_rtpg and why it's added
> 
Okay, will be doing so with the next round.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Hannes Reinecke]

On 09/01/2015 12:20 PM, Christoph Hellwig wrote:
>> +struct alua_dh_data {
>> +	struct alua_port_group	*pg;
>> +	int			rel_port;
>> +	int			tpgs;
> 
> The tpgs file doesn't make much sense in the new alua_dh_data
> structure.  Please return it explicitly from alua_check_tpgs and
> assign it directly to the member in struct alua_port_group.
> 
Okay.

>> +static void release_port_group(struct kref *kref)
>> +{
>> +	struct alua_port_group *pg;
>> +
>> +	pg = container_of(kref, struct alua_port_group, kref);
>> +	printk(KERN_WARNING "alua: release port group %d\n", pg->group_id);
> 
> Why is this a warning?  I'd consider it nothing but debug noise.
> 
Yes, indeed it is. I'll be removing it.

>> -	err = alua_rtpg(sdev, h, 0);
>> -	if (err != SCSI_DH_OK)
>> +	if (err != SCSI_DH_OK || !h->pg)
>>  		goto out;
>>  
>> +	kref_get(&h->pg->kref);
> 
> What prevents this kref_get from racing with the last kref_put?
> 
> I think all the kref_get calls in this patch need to be
> kref_get_unless_zero with proper error handling.
> 
Okay.

>> +	rcu_read_lock();
>> +	pg = rcu_dereference(h->pg);
>> +	if (!pg) {
>> +		rcu_read_unlock();
>> +		return -ENXIO;
>> +	}
> 
> What is this rcu_read_lock supposed to protect against given
> that struct alua_port_group isn't RCU freed?  I think this needs
> to be another kref_get_unless_zero.
> 
It's not freed now, but it'll be with one of the next patches (ie with
the 'rescan' patch).
I just kept it here for consistency, following the rule to always
enclose 'rcu_dereference' with 'rcu_read_lock()/rcu_read_unlock()'
pairs. Irrespective on whether it's being freed or not.

>> +
>>  	if (optimize)
>> -		h->flags |= ALUA_OPTIMIZE_STPG;
>> +		pg->flags |= ALUA_OPTIMIZE_STPG;
>>  	else
>> -		h->flags &= ~ALUA_OPTIMIZE_STPG;
>> +		pg->flags |= ~ALUA_OPTIMIZE_STPG;
>> +	rcu_read_unlock();
> 
> The read-modify-write of pg->flags will need some sort of locking.
> Seems like most modifications of it are under pg->rtpg_lock, so
> I'd suggest to enforce that as a strict rule.
> 
Yes, of course the 'rtpg_lock' should have been taken here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Christoph Hellwig]

On Tue, Sep 01, 2015 at 03:02:29PM +0200, Hannes Reinecke wrote:
> >> +	rcu_read_lock();
> >> +	pg = rcu_dereference(h->pg);
> >> +	if (!pg) {
> >> +		rcu_read_unlock();
> >> +		return -ENXIO;
> >> +	}
> > 
> > What is this rcu_read_lock supposed to protect against given
> > that struct alua_port_group isn't RCU freed?  I think this needs
> > to be another kref_get_unless_zero.
> > 
> It's not freed now, but it'll be with one of the next patches (ie with
> the 'rescan' patch).
> I just kept it here for consistency, following the rule to always
> enclose 'rcu_dereference' with 'rcu_read_lock()/rcu_read_unlock()'
> pairs. Irrespective on whether it's being freed or not.

After this patch this is the only RCU call in the driver.  I don't
think adding it here make any sense.

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Hannes Reinecke]

On 09/01/2015 03:44 PM, Christoph Hellwig wrote:
> On Tue, Sep 01, 2015 at 03:02:29PM +0200, Hannes Reinecke wrote:
>>>> +	rcu_read_lock();
>>>> +	pg = rcu_dereference(h->pg);
>>>> +	if (!pg) {
>>>> +		rcu_read_unlock();
>>>> +		return -ENXIO;
>>>> +	}
>>>
>>> What is this rcu_read_lock supposed to protect against given
>>> that struct alua_port_group isn't RCU freed?  I think this needs
>>> to be another kref_get_unless_zero.
>>>
>> It's not freed now, but it'll be with one of the next patches (ie with
>> the 'rescan' patch).
>> I just kept it here for consistency, following the rule to always
>> enclose 'rcu_dereference' with 'rcu_read_lock()/rcu_read_unlock()'
>> pairs. Irrespective on whether it's being freed or not.
> 
> After this patch this is the only RCU call in the driver.  I don't
> think adding it here make any sense.
> 
Right, I'll be moving it to the rescan patch then.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Christoph Hellwig]

On Tue, Sep 01, 2015 at 02:57:57PM +0200, Hannes Reinecke wrote:
> That is what I'm eventually planning to do.
> My final goal is to move the multipath path monitoring stuff
> into the kernel (via the block device polling mechanism), and sending
> block events for path failure and re-establishment.

It might be a good idea to prioritize that.  Todd has been asking
for multipath monitoring APIs and suggested adding D-BUS APIs to
multipathd.  I'd much prefer them directly talking to the kernel
instead of cementing multipathd, which is a bit of a wart.

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Hannes Reinecke]

On 09/02/2015 08:39 AM, Christoph Hellwig wrote:
> On Tue, Sep 01, 2015 at 02:57:57PM +0200, Hannes Reinecke wrote:
>> That is what I'm eventually planning to do.
>> My final goal is to move the multipath path monitoring stuff
>> into the kernel (via the block device polling mechanism), and sending
>> block events for path failure and re-establishment.
> 
> It might be a good idea to prioritize that.  Todd has been asking
> for multipath monitoring APIs and suggested adding D-BUS APIs to
> multipathd.  I'd much prefer them directly talking to the kernel
> instead of cementing multipathd, which is a bit of a wart.
>
Precisely my idea.
I'm fine with having a device-mapper D-Bus API, so that any application
can retrieve the device-mapper layout via D-Bus.
(It might as well be using sysfs directly, but who am I to argue here)
Path events, however, out of necessity are instantiated within the
kernel, and should be send out from the kernel via uevents.

D-Bus can be added on top of that, but multipathd should not generate
path events for D-Bus.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 01/23] scsi_dh_alua: Disable ALUA handling for non-disk devices

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Non-disk devices might support ALUA, but the firmware
Hannes> implementation is untested and frequently broken.  As we're
Hannes> don't actually need it disable ALUA support for non-disk device
Hannes> for now.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 02/23] scsi_dh_alua: Use vpd_pg83 information

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> The SCSI device now has the VPD page 0x83 information attached,
Hannes> so there is no need to query it again.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 03/23] scsi_dh_alua: improved logging

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Issue different logging messages if ALUA is not supported or the
Hannes> TPGS setting is invalid.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 06/23] scsi_dh_alua: fixup description of stpg_endio()

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Fixup copy-and-paste error in the description of stpg_endio().

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 07/23] scsi: remove scsi_show_sense_hdr()

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Last caller is gone, so remove it.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 08/23] scsi_dh_alua: use flag for RTPG extended header

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> We should be using a flag when RTPG extended header is not
Hannes> supported, that saves us sending RTPG twice for older arrays.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 09/23] scsi_dh_alua: use unaligned access macros

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Use 'get_unaligned_XX' and 'put_unaligned_XX' instead of
Hannes> open-coding it.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 10/23] scsi_dh_alua: Pass buffer as function argument

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Pass in the buffer as a function argument for submit_rtpg().

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH 01/23] scsi_dh_alua: Disable ALUA handling for non-disk devices

[Ewan Milne]

On Thu, 2015-08-27 at 14:40 +0200, Hannes Reinecke wrote:
> Non-disk devices might support ALUA, but the firmware
> implementation is untested and frequently broken.
> As we're don't actually need it disable ALUA support
> for non-disk device for now.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index cc2773b..7d01ef0 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -320,6 +320,18 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
>  	int err = SCSI_DH_OK;
>  
> +	/*
> +	 * ALUA support for non-disk devices is fraught with
> +	 * difficulties, so disable it for now.
> +	 */
> +	if (sdev->type != TYPE_DISK) {
> +		h->tpgs = TPGS_MODE_NONE;
> +		sdev_printk(KERN_INFO, sdev,
> +			    "%s: disable for non-disk devices\n",
> +			    ALUA_DH_NAME);
> +		return SCSI_DH_DEV_UNSUPP;
> +	}
> +
>  	h->tpgs = scsi_device_tpgs(sdev);
>  	switch (h->tpgs) {
>  	case TPGS_MODE_EXPLICIT|TPGS_MODE_IMPLICIT:

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 02/23] scsi_dh_alua: Use vpd_pg83 information

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> The SCSI device now has the VPD page 0x83 information attached,
> so there is no need to query it again.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 83 +++++-------------------------
>  1 file changed, 13 insertions(+), 70 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 7d01ef0..f15b977 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -131,43 +131,6 @@ static struct request *get_alua_req(struct scsi_device *sdev,
>  }
>  
>  /*
> - * submit_vpd_inquiry - Issue an INQUIRY VPD page 0x83 command
> - * @sdev: sdev the command should be sent to
> - */
> -static int submit_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
> -{
> -	struct request *rq;
> -	int err = SCSI_DH_RES_TEMP_UNAVAIL;
> -
> -	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
> -	if (!rq)
> -		goto done;
> -
> -	/* Prepare the command. */
> -	rq->cmd[0] = INQUIRY;
> -	rq->cmd[1] = 1;
> -	rq->cmd[2] = 0x83;
> -	rq->cmd[4] = h->bufflen;
> -	rq->cmd_len = COMMAND_SIZE(INQUIRY);
> -
> -	rq->sense = h->sense;
> -	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
> -	rq->sense_len = h->senselen = 0;
> -
> -	err = blk_execute_rq(rq->q, NULL, rq, 1);
> -	if (err == -EIO) {
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: evpd inquiry failed with %x\n",
> -			    ALUA_DH_NAME, rq->errors);
> -		h->senselen = rq->sense_len;
> -		err = SCSI_DH_IO;
> -	}
> -	blk_put_request(rq);
> -done:
> -	return err;
> -}
> -
> -/*
>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>   * @sdev: sdev the command should be sent to
>   */
> @@ -359,43 +322,24 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
>  }
>  
>  /*
> - * alua_vpd_inquiry - Evaluate INQUIRY vpd page 0x83
> + * alua_check_vpd - Evaluate INQUIRY vpd page 0x83
>   * @sdev: device to be checked
>   *
>   * Extract the relative target port and the target port group
>   * descriptor from the list of identificators.
>   */
> -static int alua_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
> +static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
> -	int len;
> -	unsigned err;
>  	unsigned char *d;
>  
> - retry:
> -	err = submit_vpd_inquiry(sdev, h);
> -
> -	if (err != SCSI_DH_OK)
> -		return err;
> -
> -	/* Check if vpd page exceeds initial buffer */
> -	len = (h->buff[2] << 8) + h->buff[3] + 4;
> -	if (len > h->bufflen) {
> -		/* Resubmit with the correct length */
> -		if (realloc_buffer(h, len)) {
> -			sdev_printk(KERN_WARNING, sdev,
> -				    "%s: kmalloc buffer failed\n",
> -				    ALUA_DH_NAME);
> -			/* Temporary failure, bypass */
> -			return SCSI_DH_DEV_TEMP_BUSY;
> -		}
> -		goto retry;
> -	}
> +	if (!sdev->vpd_pg83)
> +		return SCSI_DH_DEV_UNSUPP;
>  
>  	/*
> -	 * Now look for the correct descriptor.
> +	 * Look for the correct descriptor.
>  	 */
> -	d = h->buff + 4;
> -	while (d < h->buff + len) {
> +	d = sdev->vpd_pg83 + 4;
> +	while (d < sdev->vpd_pg83 + sdev->vpd_pg83_len) {
>  		switch (d[1] & 0xf) {
>  		case 0x4:
>  			/* Relative target port */
> @@ -422,14 +366,13 @@ static int alua_vpd_inquiry(struct scsi_device *sdev, struct alua_dh_data *h)
>  			    ALUA_DH_NAME);
>  		h->state = TPGS_STATE_OPTIMIZED;
>  		h->tpgs = TPGS_MODE_NONE;
> -		err = SCSI_DH_DEV_UNSUPP;
> -	} else {
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: port group %02x rel port %02x\n",
> -			    ALUA_DH_NAME, h->group_id, h->rel_port);
> +		return SCSI_DH_DEV_UNSUPP;
>  	}
> +	sdev_printk(KERN_INFO, sdev,
> +		    "%s: port group %02x rel port %02x\n",
> +		    ALUA_DH_NAME, h->group_id, h->rel_port);
>  
> -	return err;
> +	return 0;
>  }
>  
>  static char print_alua_state(int state)
> @@ -692,7 +635,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  	if (err != SCSI_DH_OK)
>  		goto out;
>  
> -	err = alua_vpd_inquiry(sdev, h);
> +	err = alua_check_vpd(sdev, h);
>  	if (err != SCSI_DH_OK)
>  		goto out;
>  

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 03/23] scsi_dh_alua: improved logging

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Issue different logging messages if ALUA is not supported
> or the TPGS setting is invalid.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index f15b977..a20c8bf 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -310,12 +310,18 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
>  		sdev_printk(KERN_INFO, sdev, "%s: supports implicit TPGS\n",
>  			    ALUA_DH_NAME);
>  		break;
> -	default:
> -		h->tpgs = TPGS_MODE_NONE;
> +	case TPGS_MODE_NONE:
>  		sdev_printk(KERN_INFO, sdev, "%s: not supported\n",
>  			    ALUA_DH_NAME);
>  		err = SCSI_DH_DEV_UNSUPP;
>  		break;
> +	default:
> +		sdev_printk(KERN_INFO, sdev,
> +			    "%s: unsupported TPGS setting %d\n",
> +			    ALUA_DH_NAME, h->tpgs);
> +		h->tpgs = TPGS_MODE_NONE;
> +		err = SCSI_DH_DEV_UNSUPP;
> +		break;
>  	}
>  
>  	return err;

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 04/23] scsi_dh_alua: use standard logging functions

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Use standard logging functions instead of hand-crafted ones.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 34 +++++++++++++-----------------
>  1 file changed, 15 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index a20c8bf..7b43ee3 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -23,6 +23,7 @@
>  #include <linux/delay.h>
>  #include <linux/module.h>
>  #include <scsi/scsi.h>
> +#include <scsi/scsi_dbg.h>
>  #include <scsi/scsi_eh.h>
>  #include <scsi/scsi_dh.h>
>  
> @@ -194,22 +195,16 @@ static void stpg_endio(struct request *req, int error)
>  		goto done;
>  	}
>  
> -	if (req->sense_len > 0) {
> -		err = scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> -					   &sense_hdr);
> -		if (!err) {
> -			err = SCSI_DH_IO;
> -			goto done;
> -		}
> +	if (scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> +				 &sense_hdr)) {
>  		err = alua_check_sense(h->sdev, &sense_hdr);
>  		if (err == ADD_TO_MLQUEUE) {
>  			err = SCSI_DH_RETRY;
>  			goto done;
>  		}
> -		sdev_printk(KERN_INFO, h->sdev,
> -			    "%s: stpg sense code: %02x/%02x/%02x\n",
> -			    ALUA_DH_NAME, sense_hdr.sense_key,
> -			    sense_hdr.asc, sense_hdr.ascq);
> +		sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
> +			    ALUA_DH_NAME);
> +		scsi_print_sense_hdr(h->sdev, ALUA_DH_NAME, &sense_hdr);
>  		err = SCSI_DH_IO;
>  	} else if (error)
>  		err = SCSI_DH_IO;
> @@ -530,16 +525,17 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		}
>  
>  		err = alua_check_sense(sdev, &sense_hdr);
> -		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry))
> +		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
> +			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
> +				    ALUA_DH_NAME);
> +			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>  			goto retry;
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: rtpg sense code %02x/%02x/%02x\n",
> -			    ALUA_DH_NAME, sense_hdr.sense_key,
> -			    sense_hdr.asc, sense_hdr.ascq);
> -		err = SCSI_DH_IO;
> +		}
> +		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
> +			    ALUA_DH_NAME);
> +		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> +		return SCSI_DH_IO;
>  	}
> -	if (err != SCSI_DH_OK)
> -		return err;
>  
>  	len = (h->buff[0] << 24) + (h->buff[1] << 16) +
>  		(h->buff[2] << 8) + h->buff[3] + 4;

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 05/23] scsi_dh_alua: return standard SCSI return codes in submit_rtpg

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Fixup submit_rtpg() to always return a standard SCSI return code.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 38 +++++++++++++++++-------------
>  1 file changed, 22 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 7b43ee3..c41d662 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -139,11 +139,13 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
>  			    bool rtpg_ext_hdr_req)
>  {
>  	struct request *rq;
> -	int err = SCSI_DH_RES_TEMP_UNAVAIL;
> +	int err = 0;
>  
>  	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
> -	if (!rq)
> +	if (!rq) {
> +		err = DRIVER_BUSY << 24;
>  		goto done;
> +	}
>  
>  	/* Prepare the command. */
>  	rq->cmd[0] = MAINTENANCE_IN;
> @@ -161,13 +163,10 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
>  	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
>  	rq->sense_len = h->senselen = 0;
>  
> -	err = blk_execute_rq(rq->q, NULL, rq, 1);
> -	if (err == -EIO) {
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: rtpg failed with %x\n",
> -			    ALUA_DH_NAME, rq->errors);
> +	blk_execute_rq(rq->q, NULL, rq, 1);
> +	if (rq->errors) {
> +		err = rq->errors;
>  		h->senselen = rq->sense_len;
> -		err = SCSI_DH_IO;
>  	}
>  	blk_put_request(rq);
>  done:
> @@ -489,7 +488,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  	struct scsi_sense_hdr sense_hdr;
>  	int len, k, off, valid_states = 0;
>  	unsigned char *ucp;
> -	unsigned err;
> +	unsigned err, retval;
>  	bool rtpg_ext_hdr_req = 1;
>  	unsigned long expiry, interval = 0;
>  	unsigned int tpg_desc_tbl_off;
> @@ -501,13 +500,20 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
>  
>   retry:
> -	err = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
> -
> -	if (err == SCSI_DH_IO && h->senselen > 0) {
> -		err = scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> -					   &sense_hdr);
> -		if (!err)
> -			return SCSI_DH_IO;
> +	retval = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
> +
> +	if (retval) {
> +		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> +					  &sense_hdr)) {
> +			sdev_printk(KERN_INFO, sdev,
> +				    "%s: rtpg failed, result %d\n",
> +				    ALUA_DH_NAME, retval);
> +			if (driver_byte(retval) == DRIVER_BUSY)
> +				err = SCSI_DH_DEV_TEMP_BUSY;
> +			else
> +				err = SCSI_DH_IO;
> +			return err;
> +		}
>  
>  		/*
>  		 * submit_rtpg() has failed on existing arrays

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 06/23] scsi_dh_alua: fixup description of stpg_endio()

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Fixup copy-and-paste error in the description of stpg_endio().
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index c41d662..25c2045 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -174,13 +174,11 @@ done:
>  }
>  
>  /*
> - * alua_stpg - Evaluate SET TARGET GROUP STATES
> + * stpg_endio - Evaluate SET TARGET GROUP STATES
>   * @sdev: the device to be evaluated
>   * @state: the new target group state
>   *
> - * Send a SET TARGET GROUP STATES command to the device.
> - * We only have to test here if we should resubmit the command;
> - * any other error is assumed as a failure.
> + * Evaluate a SET TARGET GROUP STATES command response.
>   */
>  static void stpg_endio(struct request *req, int error)
>  {

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 07/23] scsi: remove scsi_show_sense_hdr()

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Last caller is gone, so remove it.
> 
> Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Reviewed-by: Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  include/scsi/scsi_dbg.h | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/include/scsi/scsi_dbg.h b/include/scsi/scsi_dbg.h
> index f8170e9..56710e0 100644
> --- a/include/scsi/scsi_dbg.h
> +++ b/include/scsi/scsi_dbg.h
> @@ -12,8 +12,6 @@ extern size_t __scsi_format_command(char *, size_t,
>  				   const unsigned char *, size_t);
>  extern void scsi_show_extd_sense(const struct scsi_device *, const char *,
>  				 unsigned char, unsigned char);
> -extern void scsi_show_sense_hdr(const struct scsi_device *, const char *,
> -				const struct scsi_sense_hdr *);
>  extern void scsi_print_sense_hdr(const struct scsi_device *, const char *,
>  				 const struct scsi_sense_hdr *);
>  extern void scsi_print_sense(const struct scsi_cmnd *);

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 08/23] scsi_dh_alua: use flag for RTPG extended header

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> We should be using a flag when RTPG extended header is not
> supported, that saves us sending RTPG twice for older arrays.
> 
> Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 15 +++++++--------
>  1 file changed, 7 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 25c2045..9d8fc53 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -59,8 +59,9 @@
>  #define ALUA_FAILOVER_TIMEOUT		60
>  #define ALUA_FAILOVER_RETRIES		5
>  
> -/* flags passed from user level */
> +/* device handler flags */
>  #define ALUA_OPTIMIZE_STPG		1
> +#define ALUA_RTPG_EXT_HDR_UNSUPP	2
>  
>  struct alua_dh_data {
>  	int			group_id;
> @@ -135,8 +136,7 @@ static struct request *get_alua_req(struct scsi_device *sdev,
>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>   * @sdev: sdev the command should be sent to
>   */
> -static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
> -			    bool rtpg_ext_hdr_req)
> +static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
>  	struct request *rq;
>  	int err = 0;
> @@ -149,7 +149,7 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h,
>  
>  	/* Prepare the command. */
>  	rq->cmd[0] = MAINTENANCE_IN;
> -	if (rtpg_ext_hdr_req)
> +	if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP))
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
>  	else
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS;
> @@ -487,7 +487,6 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  	int len, k, off, valid_states = 0;
>  	unsigned char *ucp;
>  	unsigned err, retval;
> -	bool rtpg_ext_hdr_req = 1;
>  	unsigned long expiry, interval = 0;
>  	unsigned int tpg_desc_tbl_off;
>  	unsigned char orig_transition_tmo;
> @@ -498,7 +497,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
>  
>   retry:
> -	retval = submit_rtpg(sdev, h, rtpg_ext_hdr_req);
> +	retval = submit_rtpg(sdev, h);
>  
>  	if (retval) {
>  		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> @@ -521,10 +520,10 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		 * The retry without rtpg_ext_hdr_req set
>  		 * handles this.
>  		 */
> -		if (rtpg_ext_hdr_req == 1 &&
> +		if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
>  		    sense_hdr.sense_key == ILLEGAL_REQUEST &&
>  		    sense_hdr.asc == 0x24 && sense_hdr.ascq == 0) {
> -			rtpg_ext_hdr_req = 0;
> +			h->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
>  			goto retry;
>  		}
>  

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 09/23] scsi_dh_alua: use unaligned access macros

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Use 'get_unaligned_XX' and 'put_unaligned_XX' instead of
> open-coding it.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 23 ++++++++---------------
>  1 file changed, 8 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 9d8fc53..0636721 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -22,6 +22,7 @@
>  #include <linux/slab.h>
>  #include <linux/delay.h>
>  #include <linux/module.h>
> +#include <asm/unaligned.h>
>  #include <scsi/scsi.h>
>  #include <scsi/scsi_dbg.h>
>  #include <scsi/scsi_eh.h>
> @@ -153,10 +154,7 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
>  	else
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS;
> -	rq->cmd[6] = (h->bufflen >> 24) & 0xff;
> -	rq->cmd[7] = (h->bufflen >> 16) & 0xff;
> -	rq->cmd[8] = (h->bufflen >>  8) & 0xff;
> -	rq->cmd[9] = h->bufflen & 0xff;
> +	put_unaligned_be32(h->bufflen, &rq->cmd[6]);
>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
>  
>  	rq->sense = h->sense;
> @@ -239,8 +237,7 @@ static unsigned submit_stpg(struct alua_dh_data *h)
>  	/* Prepare the data buffer */
>  	memset(h->buff, 0, stpg_len);
>  	h->buff[4] = TPGS_STATE_OPTIMIZED & 0x0f;
> -	h->buff[6] = (h->group_id >> 8) & 0xff;
> -	h->buff[7] = h->group_id & 0xff;
> +	put_unaligned_be16(h->group_id, &h->buff[6]);
>  
>  	rq = get_alua_req(sdev, h->buff, stpg_len, WRITE);
>  	if (!rq)
> @@ -249,10 +246,7 @@ static unsigned submit_stpg(struct alua_dh_data *h)
>  	/* Prepare the command. */
>  	rq->cmd[0] = MAINTENANCE_OUT;
>  	rq->cmd[1] = MO_SET_TARGET_PGS;
> -	rq->cmd[6] = (stpg_len >> 24) & 0xff;
> -	rq->cmd[7] = (stpg_len >> 16) & 0xff;
> -	rq->cmd[8] = (stpg_len >>  8) & 0xff;
> -	rq->cmd[9] = stpg_len & 0xff;
> +	put_unaligned_be32(stpg_len, &rq->cmd[6]);
>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
>  
>  	rq->sense = h->sense;
> @@ -341,11 +335,11 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		switch (d[1] & 0xf) {
>  		case 0x4:
>  			/* Relative target port */
> -			h->rel_port = (d[6] << 8) + d[7];
> +			h->rel_port = get_unaligned_be16(&d[6]);
>  			break;
>  		case 0x5:
>  			/* Target port group */
> -			h->group_id = (d[6] << 8) + d[7];
> +			h->group_id = get_unaligned_be16(&d[6]);
>  			break;
>  		default:
>  			break;
> @@ -540,8 +534,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		return SCSI_DH_IO;
>  	}
>  
> -	len = (h->buff[0] << 24) + (h->buff[1] << 16) +
> -		(h->buff[2] << 8) + h->buff[3] + 4;
> +	len = get_unaligned_be32(&h->buff[0]) + 4;
>  
>  	if (len > h->bufflen) {
>  		/* Resubmit with the correct length */
> @@ -576,7 +569,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  	     k < len;
>  	     k += off, ucp += off) {
>  
> -		if (h->group_id == (ucp[2] << 8) + ucp[3]) {
> +		if (h->group_id == get_unaligned_be16(&ucp[2])) {
>  			h->state = ucp[0] & 0x0f;
>  			h->pref = ucp[0] >> 7;
>  			valid_states = ucp[1];

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 10/23] scsi_dh_alua: Pass buffer as function argument

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Pass in the buffer as a function argument for submit_rtpg().
> 
> Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 20 ++++++++++----------
>  1 file changed, 10 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 0636721..9e2b3af 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -137,12 +137,13 @@ static struct request *get_alua_req(struct scsi_device *sdev,
>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>   * @sdev: sdev the command should be sent to
>   */
> -static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
> +static unsigned submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
> +			    int bufflen, unsigned char *sense, int flags)
>  {
>  	struct request *rq;
>  	int err = 0;
>  
> -	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
> +	rq = get_alua_req(sdev, buff, bufflen, READ);
>  	if (!rq) {
>  		err = DRIVER_BUSY << 24;
>  		goto done;
> @@ -150,22 +151,21 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  
>  	/* Prepare the command. */
>  	rq->cmd[0] = MAINTENANCE_IN;
> -	if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP))
> +	if (!(flags & ALUA_RTPG_EXT_HDR_UNSUPP))
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
>  	else
>  		rq->cmd[1] = MI_REPORT_TARGET_PGS;
> -	put_unaligned_be32(h->bufflen, &rq->cmd[6]);
> +	put_unaligned_be32(bufflen, &rq->cmd[6]);
>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
>  
> -	rq->sense = h->sense;
> +	rq->sense = sense;
>  	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
> -	rq->sense_len = h->senselen = 0;
> +	rq->sense_len = 0;
>  
>  	blk_execute_rq(rq->q, NULL, rq, 1);
> -	if (rq->errors) {
> +	if (rq->errors)
>  		err = rq->errors;
> -		h->senselen = rq->sense_len;
> -	}
> +
>  	blk_put_request(rq);
>  done:
>  	return err;
> @@ -491,7 +491,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
>  
>   retry:
> -	retval = submit_rtpg(sdev, h);
> +	retval = submit_rtpg(sdev, h->buff, h->bufflen, h->sense, h->flags);
>  
>  	if (retval) {
>  		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,

This patch also removes the setting of h->senselen, you should mention
that in the patch description.  Or, perhaps the removal of h->senselen
should have been done as part of patch 12/23, which removed the senselen
field (and the sense buffer field) from the alua_dh_data structure.

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 11/23] scsi_dh_alua: Make stpg synchronous

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> We should be issuing STPG synchronously as we need to
> evaluate the return code on failure.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 179 +++++++++++++----------------
>  1 file changed, 83 insertions(+), 96 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 9e2b3af..fd0385e 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -172,76 +172,28 @@ done:
>  }
>  
>  /*
> - * stpg_endio - Evaluate SET TARGET GROUP STATES
> - * @sdev: the device to be evaluated
> - * @state: the new target group state
> - *
> - * Evaluate a SET TARGET GROUP STATES command response.
> - */
> -static void stpg_endio(struct request *req, int error)
> -{
> -	struct alua_dh_data *h = req->end_io_data;
> -	struct scsi_sense_hdr sense_hdr;
> -	unsigned err = SCSI_DH_OK;
> -
> -	if (host_byte(req->errors) != DID_OK ||
> -	    msg_byte(req->errors) != COMMAND_COMPLETE) {
> -		err = SCSI_DH_IO;
> -		goto done;
> -	}
> -
> -	if (scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> -				 &sense_hdr)) {
> -		err = alua_check_sense(h->sdev, &sense_hdr);
> -		if (err == ADD_TO_MLQUEUE) {
> -			err = SCSI_DH_RETRY;
> -			goto done;
> -		}
> -		sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
> -			    ALUA_DH_NAME);
> -		scsi_print_sense_hdr(h->sdev, ALUA_DH_NAME, &sense_hdr);
> -		err = SCSI_DH_IO;
> -	} else if (error)
> -		err = SCSI_DH_IO;
> -
> -	if (err == SCSI_DH_OK) {
> -		h->state = TPGS_STATE_OPTIMIZED;
> -		sdev_printk(KERN_INFO, h->sdev,
> -			    "%s: port group %02x switched to state %c\n",
> -			    ALUA_DH_NAME, h->group_id,
> -			    print_alua_state(h->state));
> -	}
> -done:
> -	req->end_io_data = NULL;
> -	__blk_put_request(req->q, req);
> -	if (h->callback_fn) {
> -		h->callback_fn(h->callback_data, err);
> -		h->callback_fn = h->callback_data = NULL;
> -	}
> -	return;
> -}
> -
> -/*
>   * submit_stpg - Issue a SET TARGET GROUP STATES command
>   *
>   * Currently we're only setting the current target port group state
>   * to 'active/optimized' and let the array firmware figure out
>   * the states of the remaining groups.
>   */
> -static unsigned submit_stpg(struct alua_dh_data *h)
> +static unsigned submit_stpg(struct scsi_device *sdev, int group_id,
> +			    unsigned char *sense)
>  {
>  	struct request *rq;
> +	unsigned char stpg_data[8];
>  	int stpg_len = 8;
> -	struct scsi_device *sdev = h->sdev;
> +	int err = 0;
>  
>  	/* Prepare the data buffer */
> -	memset(h->buff, 0, stpg_len);
> -	h->buff[4] = TPGS_STATE_OPTIMIZED & 0x0f;
> -	put_unaligned_be16(h->group_id, &h->buff[6]);
> +	memset(stpg_data, 0, stpg_len);
> +	stpg_data[4] = TPGS_STATE_OPTIMIZED & 0x0f;
> +	put_unaligned_be16(group_id, &stpg_data[6]);
>  
> -	rq = get_alua_req(sdev, h->buff, stpg_len, WRITE);
> +	rq = get_alua_req(sdev, stpg_data, stpg_len, WRITE);
>  	if (!rq)
> -		return SCSI_DH_RES_TEMP_UNAVAIL;
> +		return DRIVER_BUSY << 24;
>  
>  	/* Prepare the command. */
>  	rq->cmd[0] = MAINTENANCE_OUT;
> @@ -249,13 +201,17 @@ static unsigned submit_stpg(struct alua_dh_data *h)
>  	put_unaligned_be32(stpg_len, &rq->cmd[6]);
>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
>  
> -	rq->sense = h->sense;
> +	rq->sense = sense;
>  	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
> -	rq->sense_len = h->senselen = 0;
> -	rq->end_io_data = h;
> +	rq->sense_len = 0;
> +
> +	blk_execute_rq(rq->q, NULL, rq, 1);
> +	if (rq->errors)
> +		err = rq->errors;
> +
> +	blk_put_request(rq);
>  
> -	blk_execute_rq_nowait(rq->q, NULL, rq, 1, stpg_endio);
> -	return SCSI_DH_OK;
> +	return err;
>  }
>  
>  /*
> @@ -619,6 +575,68 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  }
>  
>  /*
> + * alua_stpg - Issue a SET TARGET GROUP STATES command
> + *
> + * Issue a SET TARGET GROUP STATES command and evaluate the
> + * response. Returns SCSI_DH_RETRY per default to trigger
> + * a re-evaluation of the target group state.
> + */
> +static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
> +{
> +	int retval;
> +	struct scsi_sense_hdr sense_hdr;
> +
> +	if (!(h->tpgs & TPGS_MODE_EXPLICIT)) {
> +		/* Only implicit ALUA supported, retry */
> +		return SCSI_DH_RETRY;
> +	}
> +	switch (h->state) {
> +	case TPGS_STATE_OPTIMIZED:
> +		return SCSI_DH_OK;
> +	case TPGS_STATE_NONOPTIMIZED:
> +		if ((h->flags & ALUA_OPTIMIZE_STPG) &&
> +		    !h->pref &&
> +		    (h->tpgs & TPGS_MODE_IMPLICIT))
> +			return SCSI_DH_OK;
> +		break;
> +	case TPGS_STATE_STANDBY:
> +	case TPGS_STATE_UNAVAILABLE:
> +		break;
> +	case TPGS_STATE_OFFLINE:
> +		return SCSI_DH_IO;
> +		break;
> +	case TPGS_STATE_TRANSITIONING:
> +		break;
> +	default:
> +		sdev_printk(KERN_INFO, sdev,
> +			    "%s: stpg failed, unhandled TPGS state %d",
> +			    ALUA_DH_NAME, h->state);
> +		return SCSI_DH_NOSYS;
> +		break;
> +	}
> +	/* Set state to transitioning */
> +	h->state = TPGS_STATE_TRANSITIONING;
> +	retval = submit_stpg(sdev, h->group_id, h->sense);
> +
> +	if (retval) {
> +		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> +					  &sense_hdr)) {
> +			sdev_printk(KERN_INFO, sdev,
> +				    "%s: stpg failed, result %d",
> +				    ALUA_DH_NAME, retval);
> +			if (driver_byte(retval) == DRIVER_BUSY)
> +				return SCSI_DH_DEV_TEMP_BUSY;
> +		} else {
> +			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
> +				    ALUA_DH_NAME);
> +			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> +		}
> +	}
> +	/* Retry RTPG */
> +	return SCSI_DH_RETRY;
> +}
> +
> +/*
>   * alua_initialize - Initialize ALUA state
>   * @sdev: the device to be initialized
>   *
> @@ -695,7 +713,6 @@ static int alua_activate(struct scsi_device *sdev,
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
>  	int err = SCSI_DH_OK;
> -	int stpg = 0;
>  
>  	err = alua_rtpg(sdev, h, 1);
>  	if (err != SCSI_DH_OK)
> @@ -704,39 +721,9 @@ static int alua_activate(struct scsi_device *sdev,
>  	if (optimize_stpg)
>  		h->flags |= ALUA_OPTIMIZE_STPG;
>  
> -	if (h->tpgs & TPGS_MODE_EXPLICIT) {
> -		switch (h->state) {
> -		case TPGS_STATE_NONOPTIMIZED:
> -			stpg = 1;
> -			if ((h->flags & ALUA_OPTIMIZE_STPG) &&
> -			    (!h->pref) &&
> -			    (h->tpgs & TPGS_MODE_IMPLICIT))
> -				stpg = 0;
> -			break;
> -		case TPGS_STATE_STANDBY:
> -		case TPGS_STATE_UNAVAILABLE:
> -			stpg = 1;
> -			break;
> -		case TPGS_STATE_OFFLINE:
> -			err = SCSI_DH_IO;
> -			break;
> -		case TPGS_STATE_TRANSITIONING:
> -			err = SCSI_DH_RETRY;
> -			break;
> -		default:
> -			break;
> -		}
> -	}
> -
> -	if (stpg) {
> -		h->callback_fn = fn;
> -		h->callback_data = data;
> -		err = submit_stpg(h);
> -		if (err == SCSI_DH_OK)
> -			return 0;
> -		h->callback_fn = h->callback_data = NULL;
> -	}
> -
> +	err = alua_stpg(sdev, h);
> +	if (err == SCSI_DH_RETRY)
> +		err = alua_rtpg(sdev, h, 1);
>  out:
>  	if (fn)
>  		fn(data, err);

submit_stpg() now returns DRIVER_BUSY << 24 instead of SCSI_DH_RES_TEMP_UNAVAIL,
so you are changing the return code semantics here as in patch 5/23.  That's
fine, but probably should be mentioned in the patch description.

The removed stpg_endio() code *did* evaluate the SCSI error code, and the code
that has replaced it doesn't exactly do the same thing, so what difference in
behavior are we going to get here?

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 12/23] scsi_dh_alua: switch to scsi_execute_req_flags()

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> All commands are issued synchronously, so no need to open-code
> scsi_execute_req_flags() anymore. And we can get rid of the
> static sense code structure element.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 124 ++++++++---------------------
>  1 file changed, 34 insertions(+), 90 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index fd0385e..ef4363a 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -75,8 +75,6 @@ struct alua_dh_data {
>  	unsigned char		*buff;
>  	int			bufflen;
>  	unsigned char		transition_tmo;
> -	unsigned char		sense[SCSI_SENSE_BUFFERSIZE];
> -	int			senselen;
>  	struct scsi_device	*sdev;
>  	activate_complete	callback_fn;
>  	void			*callback_data;
> @@ -103,72 +101,30 @@ static int realloc_buffer(struct alua_dh_data *h, unsigned len)
>  	return 0;
>  }
>  
> -static struct request *get_alua_req(struct scsi_device *sdev,
> -				    void *buffer, unsigned buflen, int rw)
> -{
> -	struct request *rq;
> -	struct request_queue *q = sdev->request_queue;
> -
> -	rq = blk_get_request(q, rw, GFP_NOIO);
> -
> -	if (IS_ERR(rq)) {
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: blk_get_request failed\n", __func__);
> -		return NULL;
> -	}
> -	blk_rq_set_block_pc(rq);
> -
> -	if (buflen && blk_rq_map_kern(q, rq, buffer, buflen, GFP_NOIO)) {
> -		blk_put_request(rq);
> -		sdev_printk(KERN_INFO, sdev,
> -			    "%s: blk_rq_map_kern failed\n", __func__);
> -		return NULL;
> -	}
> -
> -	rq->cmd_flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
> -			 REQ_FAILFAST_DRIVER;
> -	rq->retries = ALUA_FAILOVER_RETRIES;
> -	rq->timeout = ALUA_FAILOVER_TIMEOUT * HZ;
> -
> -	return rq;
> -}
> -
>  /*
>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>   * @sdev: sdev the command should be sent to
>   */
> -static unsigned submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
> -			    int bufflen, unsigned char *sense, int flags)
> +static int submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
> +		       int bufflen, struct scsi_sense_hdr *sshdr, int flags)
>  {
> -	struct request *rq;
> -	int err = 0;
> -
> -	rq = get_alua_req(sdev, buff, bufflen, READ);
> -	if (!rq) {
> -		err = DRIVER_BUSY << 24;
> -		goto done;
> -	}
> +	u8 cdb[16];

See below.

> +	int req_flags = REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
> +		REQ_FAILFAST_DRIVER;
>  
>  	/* Prepare the command. */
> -	rq->cmd[0] = MAINTENANCE_IN;
> +	memset(cdb, 0x0, 16);

See below.

> +	cdb[0] = MAINTENANCE_IN;
>  	if (!(flags & ALUA_RTPG_EXT_HDR_UNSUPP))
> -		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
> +		cdb[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
>  	else
> -		rq->cmd[1] = MI_REPORT_TARGET_PGS;
> -	put_unaligned_be32(bufflen, &rq->cmd[6]);
> -	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
> -
> -	rq->sense = sense;
> -	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
> -	rq->sense_len = 0;
> +		cdb[1] = MI_REPORT_TARGET_PGS;
> +	put_unaligned_be32(bufflen, &cdb[6]);
>  
> -	blk_execute_rq(rq->q, NULL, rq, 1);
> -	if (rq->errors)
> -		err = rq->errors;
> -
> -	blk_put_request(rq);
> -done:
> -	return err;
> +	return scsi_execute_req_flags(sdev, cdb, DMA_FROM_DEVICE,
> +				      buff, bufflen, sshdr,
> +				      ALUA_FAILOVER_TIMEOUT * HZ,
> +				      ALUA_FAILOVER_RETRIES, NULL, req_flags);
>  }
>  
>  /*
> @@ -178,40 +134,30 @@ done:
>   * to 'active/optimized' and let the array firmware figure out
>   * the states of the remaining groups.
>   */
> -static unsigned submit_stpg(struct scsi_device *sdev, int group_id,
> -			    unsigned char *sense)
> +static int submit_stpg(struct scsi_device *sdev, int group_id,
> +		       struct scsi_sense_hdr *sshdr)
>  {
> -	struct request *rq;
> +	u8 cdb[COMMAND_SIZE(MAINTENANCE_OUT)];
>  	unsigned char stpg_data[8];
>  	int stpg_len = 8;
> -	int err = 0;
> +	int req_flags = REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
> +		REQ_FAILFAST_DRIVER;
>  
>  	/* Prepare the data buffer */
>  	memset(stpg_data, 0, stpg_len);
>  	stpg_data[4] = TPGS_STATE_OPTIMIZED & 0x0f;
>  	put_unaligned_be16(group_id, &stpg_data[6]);
>  
> -	rq = get_alua_req(sdev, stpg_data, stpg_len, WRITE);
> -	if (!rq)
> -		return DRIVER_BUSY << 24;
> -
>  	/* Prepare the command. */
> -	rq->cmd[0] = MAINTENANCE_OUT;
> -	rq->cmd[1] = MO_SET_TARGET_PGS;
> -	put_unaligned_be32(stpg_len, &rq->cmd[6]);
> -	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
> -
> -	rq->sense = sense;
> -	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
> -	rq->sense_len = 0;
> -
> -	blk_execute_rq(rq->q, NULL, rq, 1);
> -	if (rq->errors)
> -		err = rq->errors;
> -
> -	blk_put_request(rq);
> -
> -	return err;
> +	memset(cdb, 0x0, COMMAND_SIZE(MAINTENANCE_OUT));

See below.

> +	cdb[0] = MAINTENANCE_OUT;
> +	cdb[1] = MO_SET_TARGET_PGS;
> +	put_unaligned_be32(stpg_len, &cdb[6]);
> +
> +	return scsi_execute_req_flags(sdev, cdb, DMA_TO_DEVICE,
> +				      stpg_data, stpg_len,
> +				      sshdr, ALUA_FAILOVER_TIMEOUT * HZ,
> +				      ALUA_FAILOVER_RETRIES, NULL, req_flags);
>  }
>  
>  /*
> @@ -447,15 +393,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
>  
>   retry:
> -	retval = submit_rtpg(sdev, h->buff, h->bufflen, h->sense, h->flags);
> +	retval = submit_rtpg(sdev, h->buff, h->bufflen, &sense_hdr, h->flags);
>  
>  	if (retval) {
> -		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> -					  &sense_hdr)) {
> +		if (!scsi_sense_valid(&sense_hdr)) {
>  			sdev_printk(KERN_INFO, sdev,
>  				    "%s: rtpg failed, result %d\n",
>  				    ALUA_DH_NAME, retval);
> -			if (driver_byte(retval) == DRIVER_BUSY)
> +			if (driver_byte(retval) == DRIVER_ERROR)
>  				err = SCSI_DH_DEV_TEMP_BUSY;
>  			else
>  				err = SCSI_DH_IO;
> @@ -616,15 +561,14 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  	}
>  	/* Set state to transitioning */
>  	h->state = TPGS_STATE_TRANSITIONING;
> -	retval = submit_stpg(sdev, h->group_id, h->sense);
> +	retval = submit_stpg(sdev, h->group_id, &sense_hdr);
>  
>  	if (retval) {
> -		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> -					  &sense_hdr)) {
> +		if (!scsi_sense_valid(&sense_hdr)) {
>  			sdev_printk(KERN_INFO, sdev,
>  				    "%s: stpg failed, result %d",
>  				    ALUA_DH_NAME, retval);
> -			if (driver_byte(retval) == DRIVER_BUSY)
> +			if (driver_byte(retval) == DRIVER_ERROR)
>  				return SCSI_DH_DEV_TEMP_BUSY;
>  		} else {
>  			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",

u8 cdb[16] and u8 cdb[COMMAND_SIZE(MAINTENANCE_OUT)] in submit_rtpg/submit_stpg
should be consistent.  I would stick with COMMAND_SIZE.  16 is wrong, RTPG is
a 12-byte MAINTENANCE_IN service action.

memset() of cdb in submit_rtpg/submit_stpg should use sizeof(cdb) rather than
a hardcoded value.

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 13/23] scsi_dh_alua: Use separate alua_port_group structure

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> The port group needs to be a separate structure as several
> LUNs might belong to the same group.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 211 +++++++++++++++++++----------
>  1 file changed, 139 insertions(+), 72 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index ef4363a..d1010dd 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -64,9 +64,13 @@
>  #define ALUA_OPTIMIZE_STPG		1
>  #define ALUA_RTPG_EXT_HDR_UNSUPP	2
>  
> -struct alua_dh_data {
> +static LIST_HEAD(port_group_list);
> +static DEFINE_SPINLOCK(port_group_lock);
> +
> +struct alua_port_group {
> +	struct kref		kref;
> +	struct list_head	node;
>  	int			group_id;
> -	int			rel_port;
>  	int			tpgs;
>  	int			state;
>  	int			pref;
> @@ -75,6 +79,12 @@ struct alua_dh_data {
>  	unsigned char		*buff;
>  	int			bufflen;
>  	unsigned char		transition_tmo;
> +};
> +
> +struct alua_dh_data {
> +	struct alua_port_group	*pg;
> +	int			rel_port;
> +	int			tpgs;
>  	struct scsi_device	*sdev;
>  	activate_complete	callback_fn;
>  	void			*callback_data;
> @@ -86,21 +96,35 @@ struct alua_dh_data {
>  static char print_alua_state(int);
>  static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
>  
> -static int realloc_buffer(struct alua_dh_data *h, unsigned len)
> +static int realloc_buffer(struct alua_port_group *pg, unsigned len)
>  {
> -	if (h->buff && h->buff != h->inq)
> -		kfree(h->buff);
> +	if (pg->buff && pg->buff != pg->inq)
> +		kfree(pg->buff);
>  
> -	h->buff = kmalloc(len, GFP_NOIO);
> -	if (!h->buff) {
> -		h->buff = h->inq;
> -		h->bufflen = ALUA_INQUIRY_SIZE;
> +	pg->buff = kmalloc(len, GFP_NOIO);
> +	if (!pg->buff) {
> +		pg->buff = pg->inq;
> +		pg->bufflen = ALUA_INQUIRY_SIZE;
>  		return 1;
>  	}
> -	h->bufflen = len;
> +	pg->bufflen = len;
>  	return 0;
>  }
>  
> +static void release_port_group(struct kref *kref)
> +{
> +	struct alua_port_group *pg;
> +
> +	pg = container_of(kref, struct alua_port_group, kref);
> +	printk(KERN_WARNING "alua: release port group %d\n", pg->group_id);
> +	spin_lock(&port_group_lock);
> +	list_del(&pg->node);
> +	spin_unlock(&port_group_lock);
> +	if (pg->buff && pg->inq != pg->buff)
> +		kfree(pg->buff);
> +	kfree(pg);
> +}
> +
>  /*
>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>   * @sdev: sdev the command should be sent to
> @@ -225,6 +249,8 @@ static int alua_check_tpgs(struct scsi_device *sdev, struct alua_dh_data *h)
>  static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
>  	unsigned char *d;
> +	int group_id = -1;
> +	struct alua_port_group *pg = NULL;
>  
>  	if (!sdev->vpd_pg83)
>  		return SCSI_DH_DEV_UNSUPP;
> @@ -241,7 +267,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  			break;
>  		case 0x5:
>  			/* Target port group */
> -			h->group_id = get_unaligned_be16(&d[6]);
> +			group_id = get_unaligned_be16(&d[6]);
>  			break;
>  		default:
>  			break;
> @@ -249,7 +275,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		d += d[3] + 4;
>  	}
>  
> -	if (h->group_id == -1) {
> +	if (group_id == -1) {
>  		/*
>  		 * Internal error; TPGS supported but required
>  		 * VPD identification descriptors not present.
> @@ -258,15 +284,33 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		sdev_printk(KERN_INFO, sdev,
>  			    "%s: No target port descriptors found\n",
>  			    ALUA_DH_NAME);
> -		h->state = TPGS_STATE_OPTIMIZED;
>  		h->tpgs = TPGS_MODE_NONE;
>  		return SCSI_DH_DEV_UNSUPP;
>  	}
>  	sdev_printk(KERN_INFO, sdev,
>  		    "%s: port group %02x rel port %02x\n",
> -		    ALUA_DH_NAME, h->group_id, h->rel_port);
> +		    ALUA_DH_NAME, group_id, h->rel_port);
>  
> -	return 0;
> +	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
> +	if (!pg) {
> +		sdev_printk(KERN_WARNING, sdev,
> +			    "%s: kzalloc port group failed\n",
> +			    ALUA_DH_NAME);
> +		/* Temporary failure, bypass */
> +		return SCSI_DH_DEV_TEMP_BUSY;
> +	}
> +	pg->group_id = group_id;
> +	pg->buff = pg->inq;
> +	pg->bufflen = ALUA_INQUIRY_SIZE;
> +	pg->tpgs = h->tpgs;
> +	pg->state = TPGS_STATE_OPTIMIZED;
> +	kref_init(&pg->kref);
> +	spin_lock(&port_group_lock);
> +	list_add(&pg->node, &port_group_list);
> +	h->pg = pg;
> +	spin_unlock(&port_group_lock);
> +
> +	return SCSI_DH_OK;
>  }
>  
>  static char print_alua_state(int state)
> @@ -377,7 +421,7 @@ static int alua_check_sense(struct scsi_device *sdev,
>   * Returns SCSI_DH_DEV_OFFLINED if the path is
>   * found to be unusable.
>   */
> -static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_for_transition)
> +static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
>  {
>  	struct scsi_sense_hdr sense_hdr;
>  	int len, k, off, valid_states = 0;
> @@ -387,13 +431,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  	unsigned int tpg_desc_tbl_off;
>  	unsigned char orig_transition_tmo;
>  
> -	if (!h->transition_tmo)
> +	if (!pg->transition_tmo)
>  		expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
>  	else
> -		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
> +		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
>  
>   retry:
> -	retval = submit_rtpg(sdev, h->buff, h->bufflen, &sense_hdr, h->flags);
> +	retval = submit_rtpg(sdev, pg->buff, pg->bufflen,
> +			     &sense_hdr, pg->flags);
>  
>  	if (retval) {
>  		if (!scsi_sense_valid(&sense_hdr)) {
> @@ -415,10 +460,10 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		 * The retry without rtpg_ext_hdr_req set
>  		 * handles this.
>  		 */
> -		if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
> +		if (!(pg->flags & ALUA_RTPG_EXT_HDR_UNSUPP) &&
>  		    sense_hdr.sense_key == ILLEGAL_REQUEST &&
>  		    sense_hdr.asc == 0x24 && sense_hdr.ascq == 0) {
> -			h->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
> +			pg->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
>  			goto retry;
>  		}
>  
> @@ -435,11 +480,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		return SCSI_DH_IO;
>  	}
>  
> -	len = get_unaligned_be32(&h->buff[0]) + 4;
> +	len = get_unaligned_be32(&pg->buff[0]) + 4;
>  
> -	if (len > h->bufflen) {
> +	if (len > pg->bufflen) {
>  		/* Resubmit with the correct length */
> -		if (realloc_buffer(h, len)) {
> +		if (realloc_buffer(pg, len)) {
>  			sdev_printk(KERN_WARNING, sdev,
>  				    "%s: kmalloc buffer failed\n",__func__);
>  			/* Temporary failure, bypass */
> @@ -448,31 +493,33 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		goto retry;
>  	}
>  
> -	orig_transition_tmo = h->transition_tmo;
> -	if ((h->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR && h->buff[5] != 0)
> -		h->transition_tmo = h->buff[5];
> +	orig_transition_tmo = pg->transition_tmo;
> +	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR &&
> +	    pg->buff[5] != 0)
> +		pg->transition_tmo = pg->buff[5];
>  	else
> -		h->transition_tmo = ALUA_FAILOVER_TIMEOUT;
> +		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
>  
> -	if (wait_for_transition && (orig_transition_tmo != h->transition_tmo)) {
> +	if (wait_for_transition &&
> +	    (orig_transition_tmo != pg->transition_tmo)) {
>  		sdev_printk(KERN_INFO, sdev,
>  			    "%s: transition timeout set to %d seconds\n",
> -			    ALUA_DH_NAME, h->transition_tmo);
> -		expiry = jiffies + h->transition_tmo * HZ;
> +			    ALUA_DH_NAME, pg->transition_tmo);
> +		expiry = jiffies + pg->transition_tmo * HZ;
>  	}
>  
> -	if ((h->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
> +	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
>  		tpg_desc_tbl_off = 8;
>  	else
>  		tpg_desc_tbl_off = 4;
>  
> -	for (k = tpg_desc_tbl_off, ucp = h->buff + tpg_desc_tbl_off;
> +	for (k = tpg_desc_tbl_off, ucp = pg->buff + tpg_desc_tbl_off;
>  	     k < len;
>  	     k += off, ucp += off) {
>  
> -		if (h->group_id == get_unaligned_be16(&ucp[2])) {
> -			h->state = ucp[0] & 0x0f;
> -			h->pref = ucp[0] >> 7;
> +		if (pg->group_id == get_unaligned_be16(&ucp[2])) {
> +			pg->state = ucp[0] & 0x0f;
> +			pg->pref = ucp[0] >> 7;
>  			valid_states = ucp[1];
>  		}
>  		off = 8 + (ucp[7] * 4);
> @@ -480,8 +527,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  
>  	sdev_printk(KERN_INFO, sdev,
>  		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
> -		    ALUA_DH_NAME, h->group_id, print_alua_state(h->state),
> -		    h->pref ? "preferred" : "non-preferred",
> +		    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),
> +		    pg->pref ? "preferred" : "non-preferred",
>  		    valid_states&TPGS_SUPPORT_TRANSITION?'T':'t',
>  		    valid_states&TPGS_SUPPORT_OFFLINE?'O':'o',
>  		    valid_states&TPGS_SUPPORT_LBA_DEPENDENT?'L':'l',
> @@ -490,7 +537,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		    valid_states&TPGS_SUPPORT_NONOPTIMIZED?'N':'n',
>  		    valid_states&TPGS_SUPPORT_OPTIMIZED?'A':'a');
>  
> -	switch (h->state) {
> +	switch (pg->state) {
>  	case TPGS_STATE_TRANSITIONING:
>  		if (wait_for_transition) {
>  			if (time_before(jiffies, expiry)) {
> @@ -505,7 +552,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>  		}
>  
>  		/* Transitioning time exceeded, set port to standby */
> -		h->state = TPGS_STATE_STANDBY;
> +		pg->state = TPGS_STATE_STANDBY;
>  		break;
>  	case TPGS_STATE_OFFLINE:
>  		/* Path unusable */
> @@ -526,22 +573,22 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>   * response. Returns SCSI_DH_RETRY per default to trigger
>   * a re-evaluation of the target group state.
>   */
> -static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
> +static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  {
>  	int retval;
>  	struct scsi_sense_hdr sense_hdr;
>  
> -	if (!(h->tpgs & TPGS_MODE_EXPLICIT)) {
> +	if (!(pg->tpgs & TPGS_MODE_EXPLICIT)) {
>  		/* Only implicit ALUA supported, retry */
>  		return SCSI_DH_RETRY;
>  	}
> -	switch (h->state) {
> +	switch (pg->state) {
>  	case TPGS_STATE_OPTIMIZED:
>  		return SCSI_DH_OK;
>  	case TPGS_STATE_NONOPTIMIZED:
> -		if ((h->flags & ALUA_OPTIMIZE_STPG) &&
> -		    !h->pref &&
> -		    (h->tpgs & TPGS_MODE_IMPLICIT))
> +		if ((pg->flags & ALUA_OPTIMIZE_STPG) &&
> +		    !pg->pref &&
> +		    (pg->tpgs & TPGS_MODE_IMPLICIT))
>  			return SCSI_DH_OK;
>  		break;
>  	case TPGS_STATE_STANDBY:
> @@ -555,13 +602,13 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  	default:
>  		sdev_printk(KERN_INFO, sdev,
>  			    "%s: stpg failed, unhandled TPGS state %d",
> -			    ALUA_DH_NAME, h->state);
> +			    ALUA_DH_NAME, pg->state);
>  		return SCSI_DH_NOSYS;
>  		break;
>  	}
>  	/* Set state to transitioning */
> -	h->state = TPGS_STATE_TRANSITIONING;
> -	retval = submit_stpg(sdev, h->group_id, &sense_hdr);
> +	pg->state = TPGS_STATE_TRANSITIONING;
> +	retval = submit_stpg(sdev, pg->group_id, &sense_hdr);
>  
>  	if (retval) {
>  		if (!scsi_sense_valid(&sense_hdr)) {
> @@ -571,7 +618,7 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
>  			if (driver_byte(retval) == DRIVER_ERROR)
>  				return SCSI_DH_DEV_TEMP_BUSY;
>  		} else {
> -			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
> +			sdev_printk(KERN_INFO, sdev, "%s: stpg failed\n",
>  				    ALUA_DH_NAME);
>  			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>  		}
> @@ -596,13 +643,12 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  		goto out;
>  
>  	err = alua_check_vpd(sdev, h);
> -	if (err != SCSI_DH_OK)
> -		goto out;
> -
> -	err = alua_rtpg(sdev, h, 0);
> -	if (err != SCSI_DH_OK)
> +	if (err != SCSI_DH_OK || !h->pg)
>  		goto out;
>  
> +	kref_get(&h->pg->kref);
> +	err = alua_rtpg(sdev, h->pg, 0);
> +	kref_put(&h->pg->kref, release_port_group);
>  out:
>  	return err;
>  }
> @@ -618,6 +664,7 @@ out:
>  static int alua_set_params(struct scsi_device *sdev, const char *params)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
> +	struct alua_port_group *pg = NULL;
>  	unsigned int optimize = 0, argc;
>  	const char *p = params;
>  	int result = SCSI_DH_OK;
> @@ -630,10 +677,18 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
>  	if ((sscanf(p, "%u", &optimize) != 1) || (optimize > 1))
>  		return -EINVAL;
>  
> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (!pg) {
> +		rcu_read_unlock();
> +		return -ENXIO;
> +	}
> +
>  	if (optimize)
> -		h->flags |= ALUA_OPTIMIZE_STPG;
> +		pg->flags |= ALUA_OPTIMIZE_STPG;
>  	else
> -		h->flags &= ~ALUA_OPTIMIZE_STPG;
> +		pg->flags |= ~ALUA_OPTIMIZE_STPG;
> +	rcu_read_unlock();
>  
>  	return result;
>  }
> @@ -658,16 +713,23 @@ static int alua_activate(struct scsi_device *sdev,
>  	struct alua_dh_data *h = sdev->handler_data;
>  	int err = SCSI_DH_OK;
>  
> -	err = alua_rtpg(sdev, h, 1);
> -	if (err != SCSI_DH_OK)
> +	if (!h->pg)
>  		goto out;
>  
> +	kref_get(&h->pg->kref);
> +
>  	if (optimize_stpg)
> -		h->flags |= ALUA_OPTIMIZE_STPG;
> +		h->pg->flags |= ALUA_OPTIMIZE_STPG;
>  
> -	err = alua_stpg(sdev, h);
> +	err = alua_rtpg(sdev, h->pg, 1);
> +	if (err != SCSI_DH_OK) {
> +		kref_put(&h->pg->kref, release_port_group);
> +		goto out;
> +	}
> +	err = alua_stpg(sdev, h->pg);
>  	if (err == SCSI_DH_RETRY)
> -		err = alua_rtpg(sdev, h, 1);
> +		err = alua_rtpg(sdev, h->pg, 1);
> +	kref_put(&h->pg->kref, release_port_group);
>  out:
>  	if (fn)
>  		fn(data, err);
> @@ -683,13 +745,19 @@ out:
>  static int alua_prep_fn(struct scsi_device *sdev, struct request *req)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
> +	int state;
>  	int ret = BLKPREP_OK;
>  
> -	if (h->state == TPGS_STATE_TRANSITIONING)
> +	if (!h->pg)
> +		return ret;
> +	kref_get(&h->pg->kref);
> +	state = h->pg->state;
> +	kref_put(&h->pg->kref, release_port_group);
> +	if (state == TPGS_STATE_TRANSITIONING)
>  		ret = BLKPREP_DEFER;
> -	else if (h->state != TPGS_STATE_OPTIMIZED &&
> -		 h->state != TPGS_STATE_NONOPTIMIZED &&
> -		 h->state != TPGS_STATE_LBA_DEPENDENT) {
> +	else if (state != TPGS_STATE_OPTIMIZED &&
> +		 state != TPGS_STATE_NONOPTIMIZED &&
> +		 state != TPGS_STATE_LBA_DEPENDENT) {
>  		ret = BLKPREP_KILL;
>  		req->cmd_flags |= REQ_QUIET;
>  	}
> @@ -710,11 +778,8 @@ static int alua_bus_attach(struct scsi_device *sdev)
>  	if (!h)
>  		return -ENOMEM;
>  	h->tpgs = TPGS_MODE_UNINITIALIZED;
> -	h->state = TPGS_STATE_OPTIMIZED;
> -	h->group_id = -1;
> +	h->pg = NULL;
>  	h->rel_port = -1;
> -	h->buff = h->inq;
> -	h->bufflen = ALUA_INQUIRY_SIZE;
>  	h->sdev = sdev;
>  
>  	err = alua_initialize(sdev, h);
> @@ -736,8 +801,10 @@ static void alua_bus_detach(struct scsi_device *sdev)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
>  
> -	if (h->buff && h->inq != h->buff)
> -		kfree(h->buff);
> +	if (h->pg) {
> +		kref_put(&h->pg->kref, release_port_group);
> +		h->pg = NULL;
> +	}
>  	sdev->handler_data = NULL;
>  	kfree(h);
>  }

#include <linux/rcupdate.h> and RCU write side locking is not added until patch 19/23.
This patch uses rcu_read_lock()/rcu_read_unlock()/rcu_dereference() and should have
the #include added.

Note:  lookup of existing alua_port_group object is not added until patch 17/23.

Note that in alua_check_vpd(), returning SCSI_DH_DEV_TEMP_BUSY if kmalloc() or pg
or alloc_ordered_workqueue() fails will cause alua_bus_attach to fail with -EINVAL.
i.e. "temp busy" will not get retried so this is kind of misleading.

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 14/23] scsi_dh_alua: allocate RTPG buffer separately

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> The RTPG buffer will only evaluated within alua_rtpg(),
> so we can allocate it locally there and avoid having to
> put it into the global structure.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 56 +++++++++++-------------------
>  1 file changed, 21 insertions(+), 35 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index d1010dd..4157fe2 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -56,7 +56,7 @@
>  #define TPGS_MODE_IMPLICIT		0x1
>  #define TPGS_MODE_EXPLICIT		0x2
>  
> -#define ALUA_INQUIRY_SIZE		36
> +#define ALUA_RTPG_SIZE			128
>  #define ALUA_FAILOVER_TIMEOUT		60
>  #define ALUA_FAILOVER_RETRIES		5
>  
> @@ -75,9 +75,6 @@ struct alua_port_group {
>  	int			state;
>  	int			pref;
>  	unsigned		flags; /* used for optimizing STPG */
> -	unsigned char		inq[ALUA_INQUIRY_SIZE];
> -	unsigned char		*buff;
> -	int			bufflen;
>  	unsigned char		transition_tmo;
>  };
>  
> @@ -96,21 +93,6 @@ struct alua_dh_data {
>  static char print_alua_state(int);
>  static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
>  
> -static int realloc_buffer(struct alua_port_group *pg, unsigned len)
> -{
> -	if (pg->buff && pg->buff != pg->inq)
> -		kfree(pg->buff);
> -
> -	pg->buff = kmalloc(len, GFP_NOIO);
> -	if (!pg->buff) {
> -		pg->buff = pg->inq;
> -		pg->bufflen = ALUA_INQUIRY_SIZE;
> -		return 1;
> -	}
> -	pg->bufflen = len;
> -	return 0;
> -}
> -
>  static void release_port_group(struct kref *kref)
>  {
>  	struct alua_port_group *pg;
> @@ -120,8 +102,6 @@ static void release_port_group(struct kref *kref)
>  	spin_lock(&port_group_lock);
>  	list_del(&pg->node);
>  	spin_unlock(&port_group_lock);
> -	if (pg->buff && pg->inq != pg->buff)
> -		kfree(pg->buff);
>  	kfree(pg);
>  }
>  
> @@ -300,8 +280,6 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		return SCSI_DH_DEV_TEMP_BUSY;
>  	}
>  	pg->group_id = group_id;
> -	pg->buff = pg->inq;
> -	pg->bufflen = ALUA_INQUIRY_SIZE;
>  	pg->tpgs = h->tpgs;
>  	pg->state = TPGS_STATE_OPTIMIZED;
>  	kref_init(&pg->kref);
> @@ -424,8 +402,8 @@ static int alua_check_sense(struct scsi_device *sdev,
>  static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
>  {
>  	struct scsi_sense_hdr sense_hdr;
> -	int len, k, off, valid_states = 0;
> -	unsigned char *ucp;
> +	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
> +	unsigned char *ucp, *buff;
>  	unsigned err, retval;
>  	unsigned long expiry, interval = 0;
>  	unsigned int tpg_desc_tbl_off;
> @@ -436,9 +414,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  	else
>  		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
>  
> +	buff = kzalloc(bufflen, GFP_KERNEL);
> +	if (!buff)
> +		return SCSI_DH_DEV_TEMP_BUSY;
> +
>   retry:
> -	retval = submit_rtpg(sdev, pg->buff, pg->bufflen,
> -			     &sense_hdr, pg->flags);
> +	retval = submit_rtpg(sdev, buff, bufflen, &sense_hdr, pg->flags);
>  
>  	if (retval) {
>  		if (!scsi_sense_valid(&sense_hdr)) {
> @@ -449,6 +430,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  				err = SCSI_DH_DEV_TEMP_BUSY;
>  			else
>  				err = SCSI_DH_IO;
> +			kfree(buff);
>  			return err;
>  		}
>  
> @@ -477,14 +459,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
>  			    ALUA_DH_NAME);
>  		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> +		kfree(buff);
>  		return SCSI_DH_IO;
>  	}
>  
> -	len = get_unaligned_be32(&pg->buff[0]) + 4;
> +	len = get_unaligned_be32(&buff[0]) + 4;
>  
> -	if (len > pg->bufflen) {
> +	if (len > bufflen) {
>  		/* Resubmit with the correct length */
> -		if (realloc_buffer(pg, len)) {
> +		kfree(buff);
> +		bufflen = len;
> +		buff = kmalloc(bufflen, GFP_KERNEL);
> +		if (!buff) {
>  			sdev_printk(KERN_WARNING, sdev,
>  				    "%s: kmalloc buffer failed\n",__func__);
>  			/* Temporary failure, bypass */
> @@ -494,9 +480,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  	}
>  
>  	orig_transition_tmo = pg->transition_tmo;
> -	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR &&
> -	    pg->buff[5] != 0)
> -		pg->transition_tmo = pg->buff[5];
> +	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR && buff[5] != 0)
> +		pg->transition_tmo = buff[5];
>  	else
>  		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
>  
> @@ -508,12 +493,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  		expiry = jiffies + pg->transition_tmo * HZ;
>  	}
>  
> -	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
> +	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
>  		tpg_desc_tbl_off = 8;
>  	else
>  		tpg_desc_tbl_off = 4;
>  
> -	for (k = tpg_desc_tbl_off, ucp = pg->buff + tpg_desc_tbl_off;
> +	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
>  	     k < len;
>  	     k += off, ucp += off) {
>  
> @@ -563,6 +548,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  		err = SCSI_DH_OK;
>  		break;
>  	}
> +	kfree(buff);
>  	return err;
>  }
>  

I guess.  Are these buffers so big that we don't want to make it part of
a structure that just stays allocated?

We should have some validation on the length of the data returned by RTPG in
the case where the buffer is re-kmalloc'ed, we can't kmalloc 4GB - 1 + 4 bytes.
And if the length is too big, it is not a "temporary" failure.
We should maybe only retry once with a larger buffer, rather than potentially
loop forever if the device keeps returning different information.

(I suppose we have the same potential issue with REPORT LUNS as well.)

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 15/23] scsi_dh_alua: simplify sense code handling

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Most sense code is already handled in the generic
> code, so we shouldn't be adding special cases here.
> However, when doing so we need to check for
> unit attention whenever we're sending an internal
> command.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 50 +++++++-----------------------
>  1 file changed, 11 insertions(+), 39 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 4157fe2..dbe9ff2 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -91,7 +91,6 @@ struct alua_dh_data {
>  #define ALUA_POLICY_SWITCH_ALL		1
>  
>  static char print_alua_state(int);
> -static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
>  
>  static void release_port_group(struct kref *kref)
>  {
> @@ -323,28 +322,6 @@ static int alua_check_sense(struct scsi_device *sdev,
>  			 * LUN Not Accessible - ALUA state transition
>  			 */
>  			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0b)
> -			/*
> -			 * LUN Not Accessible -- Target port in standby state
> -			 */
> -			return SUCCESS;
> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0c)
> -			/*
> -			 * LUN Not Accessible -- Target port in unavailable state
> -			 */
> -			return SUCCESS;
> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x12)
> -			/*
> -			 * LUN Not Ready -- Offline
> -			 */
> -			return SUCCESS;
> -		if (sdev->allow_restart &&
> -		    sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x02)
> -			/*
> -			 * if the device is not started, we need to wake
> -			 * the error handler to start the motor
> -			 */
> -			return FAILED;
>  		break;
>  	case UNIT_ATTENTION:
>  		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
> @@ -359,7 +336,7 @@ static int alua_check_sense(struct scsi_device *sdev,
>  			return ADD_TO_MLQUEUE;
>  		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
>  			/*
> -			 * Mode Parameters Changed
> +			 * Mode Parameter Changed

See below.

>  			 */
>  			return ADD_TO_MLQUEUE;
>  		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
> @@ -372,18 +349,6 @@ static int alua_check_sense(struct scsi_device *sdev,
>  			 * Implicit ALUA state transition failed
>  			 */
>  			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x03)
> -			/*
> -			 * Inquiry data has changed
> -			 */
> -			return ADD_TO_MLQUEUE;

??? See below.

> -		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x0e)
> -			/*
> -			 * REPORTED_LUNS_DATA_HAS_CHANGED is reported
> -			 * when switching controllers on targets like
> -			 * Intel Multi-Flex. We can just retry.
> -			 */
> -			return ADD_TO_MLQUEUE;

??? See below.

>  		break;
>  	}
>  
> @@ -448,9 +413,16 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  			pg->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
>  			goto retry;
>  		}
> -
> -		err = alua_check_sense(sdev, &sense_hdr);
> -		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
> +		/*
> +		 * Retry on ALUA state transition or if any
> +		 * UNIT ATTENTION occurred.
> +		 */
> +		if (sense_hdr.sense_key == NOT_READY &&
> +		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
> +			err = SCSI_DH_RETRY;
> +		else if (sense_hdr.sense_key == UNIT_ATTENTION)
> +			err = SCSI_DH_RETRY;
> +		if (err == SCSI_DH_RETRY && time_before(jiffies, expiry)) {
>  			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
>  				    ALUA_DH_NAME);
>  			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);

"Mode Parameters Changed" comment should not have been changed to "Mode Parameter Changed".
The actual T10 text is "Mode Parameters Changed", so leave it the way it is.

The sense code handling in the UA case for ASC/ASCQ 3F 03 is changed by this patch to
return SUCCESS from scsi_check_sense() instead of ADD_TO_MLQUEUE from alua_check_sense(),
and the sense code handling in the UA case for ASC/ASCQ 3F 0E is changed by this patch to
return NEEDS_RETRY from scsi_check_sense() instead of ADD_TO_MLQUEUE from alua_check_sense().
So we will not reissue the command on INQUIRY DATA HAS CHANGED and will have different
flow of control on REPORTED LUNS DATA HAS CHANGED.  What is the reason for this?

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 16/23] scsi: Add scsi_vpd_lun_id()

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Add a function scsi_vpd_lun_id() to return a unique device
> identifcation based on the designation descriptors of
> VPD page 0x83.
> 
> As devices might implement several descriptors the order
> of preference is:
> - NAA IEE Registered Extended
> - EUI-64 based 16-byte
> - EUI-64 based 12-byte
> - NAA IEEE Registered
> - NAA IEEE Extended
> A SCSI name string descriptor is preferred to all of them
> if the identification is longer than 16 bytes.
> 
> The returned unique device identification will be formatted
> as a SCSI Name string to avoid clashes between different
> designator types.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/scsi_lib.c    | 121 +++++++++++++++++++++++++++++++++++++++++++++
>  include/scsi/scsi_device.h |   1 +
>  2 files changed, 122 insertions(+)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 2a864b8..48d6ff6 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -3145,3 +3145,124 @@ void sdev_enable_disk_events(struct scsi_device *sdev)
>  	atomic_dec(&sdev->disk_events_disable_depth);
>  }
>  EXPORT_SYMBOL(sdev_enable_disk_events);
> +
> +/*
> + * scsi_vpd_lun_id - return a unique device identification
> + * @sdev: SCSI device
> + * @id:   buffer for the identification
> + * @id_len:  length of the buffer
> + *
> + * Copies a unique device identification into @id based
> + * on the information in the VPD page 0x83 of the device.
> + * The string will be formatted as a SCSI name string.
> + *
> + * Returns the length of the identification or error on failure.
> + */
> +int scsi_vpd_lun_id(struct scsi_device *sdev, char *id, size_t id_len)
> +{
> +	u8 cur_id_type = 0xff;
> +	u8 cur_id_size = 0;
> +	unsigned char *d, *cur_id_str;
> +	int id_size = -EAGAIN;
> +
> +	/*
> +	 * Look for the correct descriptor.
> +	 * Order of preference for lun descriptor:
> +	 * - SCSI name string
> +	 * - NAA IEEE Registered Extended
> +	 * - EUI-64 based 16-byte
> +	 * - EUI-64 based 12-byte
> +	 * - NAA IEEE Registered
> +	 * - NAA IEEE Extended
> +	 * as longer descriptors reduce the likelyhood
> +	 * of identification clashes.
> +	 */
> +
> +	/* The id string must be at least 20 bytes + terminating NULL byte */
> +	if (id_len < 21)
> +		return -EINVAL;
> +
> +	memset(id, 0, id_len);
> +	d = sdev->vpd_pg83 + 4;
> +	while (d < sdev->vpd_pg83 + sdev->vpd_pg83_len) {
> +		/* Skip designators not referring to the LUN */
> +		if ((d[1] & 0x30) != 0x00)
> +			goto next_desig;
> +
> +		switch (d[1] & 0xf) {
> +		case 0x2:
> +			/* EUI-64 */
> +			if (cur_id_size > d[3])
> +				break;
> +			/* Prefer NAA IEEE Registered Extended */
> +			if (cur_id_type == 0x3 &&
> +			    cur_id_size == d[3])
> +				break;
> +			cur_id_size = d[3];
> +			cur_id_str = d + 4;
> +			cur_id_type = d[1] & 0xf;
> +			switch (cur_id_size) {
> +			case 8:
> +				id_size = snprintf(id, id_len,
> +						   "eui.%8phN",
> +						   cur_id_str);
> +				break;
> +			case 12:
> +				id_size = snprintf(id, id_len,
> +						   "eui.%12phN",
> +						   cur_id_str);
> +				break;
> +			case 16:
> +				id_size = snprintf(id, id_len,
> +						   "eui.%16phN",
> +						   cur_id_str);
> +				break;
> +			default:
> +				cur_id_size = 0;
> +				break;
> +			}
> +			break;
> +		case 0x3:
> +			/* NAA */
> +			if (cur_id_size > d[3])
> +				break;
> +			cur_id_size = d[3];
> +			cur_id_str = d + 4;
> +			cur_id_type = d[1] & 0xf;
> +			switch (cur_id_size) {
> +			case 8:
> +				id_size = snprintf(id, id_len,
> +						   "naa.%8phN",
> +						   cur_id_str);
> +				break;
> +			case 16:
> +				id_size = snprintf(id, id_len,
> +						   "naa.%16phN",
> +						   cur_id_str);
> +				break;
> +			default:
> +				cur_id_size = 0;
> +				break;
> +			}
> +			break;
> +		case 0x8:
> +			/* SCSI name string */
> +			if (cur_id_size + 4 > d[3])
> +				break;
> +			cur_id_size = d[3];
> +			cur_id_str = d + 4;
> +			cur_id_type = d[1] & 0xf;
> +			if (cur_id_size >= id_len)
> +				cur_id_size = id_len - 1;
> +			memcpy(id, cur_id_str, cur_id_size);
> +			break;
> +		default:
> +			break;
> +		}
> +next_desig:
> +		d += d[3] + 4;
> +	}
> +
> +	return id_size;
> +}
> +EXPORT_SYMBOL(scsi_vpd_lun_id);
> diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
> index b6b1dc4..595a9be 100644
> --- a/include/scsi/scsi_device.h
> +++ b/include/scsi/scsi_device.h
> @@ -413,6 +413,7 @@ static inline int scsi_execute_req(struct scsi_device *sdev,
>  }
>  extern void sdev_disable_disk_events(struct scsi_device *sdev);
>  extern void sdev_enable_disk_events(struct scsi_device *sdev);
> +extern int scsi_vpd_lun_id(struct scsi_device *, char *, size_t);
>  
>  #ifdef CONFIG_PM
>  extern int scsi_autopm_get_device(struct scsi_device *);

scsi_vpd_lun_id() is an exported function, but does not check if sdev->vpd_pg83 != NULL
(i.e. no VPD 83 info), and we will crash if it is NULL.  Also, why return -EAGAIN?
What will be different next time?  Maybe a different errno would be better, like
-ENOENT or -ENODATA, I'm not sure.

In the SCSI name string case, if the name has to be truncated to fit the supplied buffer,
scsi_vpd_lun_id() does not ensure that the last byte is zero, it relies on the caller to
have done this, (which alua_check_vpd() does not do).  I'm not sure if we should truncate
the SCSI name string here or return an error here, as a truncated name will likely not
be unique.

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 17/23] scsi_dh_alua: use unique device id

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Use scsi_vpd_lun_id() to assign a unique device identification
> to the alua port group structure.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 70 +++++++++++++++++++++++++++---
>  1 file changed, 65 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index dbe9ff2..c2b2100b 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -70,6 +70,8 @@ static DEFINE_SPINLOCK(port_group_lock);
>  struct alua_port_group {
>  	struct kref		kref;
>  	struct list_head	node;
> +	unsigned char		device_id_str[256];
> +	int			device_id_size;

I prefer _len instead of _size, _size should refer to the size of the buffer,
not the current length of the data in it.

>  	int			group_id;
>  	int			tpgs;
>  	int			state;
> @@ -229,7 +231,9 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
>  	unsigned char *d;
>  	int group_id = -1;
> -	struct alua_port_group *pg = NULL;
> +	char device_id_str[256];
> +	int device_id_size;
> +	struct alua_port_group *tmp_pg, *pg = NULL;
>  
>  	if (!sdev->vpd_pg83)
>  		return SCSI_DH_DEV_UNSUPP;
> @@ -266,9 +270,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		h->tpgs = TPGS_MODE_NONE;
>  		return SCSI_DH_DEV_UNSUPP;
>  	}
> +	device_id_size = scsi_vpd_lun_id(sdev, device_id_str, 256);

should be sizeof(device_id_str) not hardcoded 256

> +	if (device_id_size <= 0) {
> +		/*
> +		 * Internal error: TPGS supported by no

"Internal error: TPGS supported by no"  should be "but no"

> +		 * device identifcation found.
> +		 * Disable ALUA support.
> +		 */
> +		sdev_printk(KERN_INFO, sdev,
> +			    "%s: No device descriptors found\n",
> +			    ALUA_DH_NAME);
> +		h->tpgs = TPGS_MODE_NONE;
> +		return SCSI_DH_DEV_UNSUPP;
> +	}
>  	sdev_printk(KERN_INFO, sdev,
> -		    "%s: port group %02x rel port %02x\n",
> -		    ALUA_DH_NAME, group_id, h->rel_port);
> +		    "%s: device %s port group %02x "
> +		    "rel port %02x\n", ALUA_DH_NAME,
> +		    device_id_str, group_id, h->rel_port);
> +	spin_lock(&port_group_lock);
> +	list_for_each_entry(tmp_pg, &port_group_list, node) {
> +		if (tmp_pg->group_id != group_id)
> +			continue;
> +		if (tmp_pg->device_id_size != device_id_size)
> +			continue;
> +		if (strncmp(tmp_pg->device_id_str, device_id_str,
> +			    device_id_size))
> +			continue;
> +		h->pg = tmp_pg;
> +		kref_get(&tmp_pg->kref);
> +		break;
> +	}
> +	spin_unlock(&port_group_lock);
> +	if (h->pg)
> +		return SCSI_DH_OK;

The lookup checks whether h->pg == NULL but the function never
explicitly sets it to NULL before iterating.

>  
>  	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
>  	if (!pg) {
> @@ -278,13 +312,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		/* Temporary failure, bypass */
>  		return SCSI_DH_DEV_TEMP_BUSY;
>  	}
> +	if (device_id_size)
> +		strncpy(pg->device_id_str, device_id_str, 256);

should be sizeof(device_id_str) not hardcoded 256

> +	else
> +		pg->device_id_str[0] = '\0';
> +
> +	pg->device_id_size = device_id_size;
>  	pg->group_id = group_id;
>  	pg->tpgs = h->tpgs;
>  	pg->state = TPGS_STATE_OPTIMIZED;
>  	kref_init(&pg->kref);
>  	spin_lock(&port_group_lock);
> -	list_add(&pg->node, &port_group_list);
> -	h->pg = pg;
> +	/*
> +	 * Re-check list again to catch
> +	 * concurrent updates
> +	 */
> +	list_for_each_entry(tmp_pg, &port_group_list, node) {
> +		if (tmp_pg->group_id != pg->group_id)
> +			continue;
> +		if (tmp_pg->device_id_size != pg->device_id_size)
> +			continue;
> +		if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
> +			    device_id_size))
> +			continue;
> +		h->pg = tmp_pg;
> +		kref_get(&tmp_pg->kref);
> +		kfree(pg);

With the added check for an existing alua_port_group object, and the kfree() of
the alua_port_group that had been allocated if an existing one is found, the code does not
do a destroy_workqueue() on pg->work_q.  

> +		pg = NULL;
> +		break;
> +	}
> +	if (pg) {
> +		list_add(&pg->node, &port_group_list);
> +		h->pg = pg;
> +	}
>  	spin_unlock(&port_group_lock);
>  
>  	return SCSI_DH_OK;
 
An explanation in the comments about the ALUA topology and what the device_id vs.
the group_id represents might be helpful.  It occurred to me that if someone doesn't
understand that you can have the same device_id behind different port groups they
won't understand this code.

HOST  ---->     STORAGE CTRL PG 1   ---->    LUN X
      ---->                         ---->    LUN X
      ---->     STORAGE CTRL PG 2   ---->    LUN X
      ---->                         ---->    LUN X

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 18/23] revert "scsi_dh_alua: ALUA hander attach should succeed while TPG is transitioning"

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> This reverts commit a8e5a2d593cbfccf530c3382c2c328d2edaa7b66
> 
> Obsoleted by the next patch.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 29 +++++++++++------------------
>  1 file changed, 11 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index c2b2100b..b52db8b 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -418,13 +418,12 @@ static int alua_check_sense(struct scsi_device *sdev,
>  /*
>   * alua_rtpg - Evaluate REPORT TARGET GROUP STATES
>   * @sdev: the device to be evaluated.
> - * @wait_for_transition: if nonzero, wait ALUA_FAILOVER_TIMEOUT seconds for device to exit transitioning state
>   *
>   * Evaluate the Target Port Group State.
>   * Returns SCSI_DH_DEV_OFFLINED if the path is
>   * found to be unusable.
>   */
> -static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
> +static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  {
>  	struct scsi_sense_hdr sense_hdr;
>  	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
> @@ -517,8 +516,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  	else
>  		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
>  
> -	if (wait_for_transition &&
> -	    (orig_transition_tmo != pg->transition_tmo)) {
> +	if (orig_transition_tmo != pg->transition_tmo) {
>  		sdev_printk(KERN_INFO, sdev,
>  			    "%s: transition timeout set to %d seconds\n",
>  			    ALUA_DH_NAME, pg->transition_tmo);
> @@ -556,19 +554,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>  
>  	switch (pg->state) {
>  	case TPGS_STATE_TRANSITIONING:
> -		if (wait_for_transition) {
> -			if (time_before(jiffies, expiry)) {
> -				/* State transition, retry */
> -				interval += 2000;
> -				msleep(interval);
> -				goto retry;
> -			}
> -			err = SCSI_DH_RETRY;
> -		} else {
> -			err = SCSI_DH_OK;
> +		if (time_before(jiffies, expiry)) {
> +			/* State transition, retry */
> +			interval += 2000;
> +			msleep(interval);
> +			goto retry;
>  		}
> -
>  		/* Transitioning time exceeded, set port to standby */
> +		err = SCSI_DH_RETRY;
>  		pg->state = TPGS_STATE_STANDBY;
>  		break;
>  	case TPGS_STATE_OFFLINE:
> @@ -665,7 +658,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  		goto out;
>  
>  	kref_get(&h->pg->kref);
> -	err = alua_rtpg(sdev, h->pg, 0);
> +	err = alua_rtpg(sdev, h->pg);
>  	kref_put(&h->pg->kref, release_port_group);
>  out:
>  	return err;
> @@ -739,14 +732,14 @@ static int alua_activate(struct scsi_device *sdev,
>  	if (optimize_stpg)
>  		h->pg->flags |= ALUA_OPTIMIZE_STPG;
>  
> -	err = alua_rtpg(sdev, h->pg, 1);
> +	err = alua_rtpg(sdev, h->pg);
>  	if (err != SCSI_DH_OK) {
>  		kref_put(&h->pg->kref, release_port_group);
>  		goto out;
>  	}
>  	err = alua_stpg(sdev, h->pg);
>  	if (err == SCSI_DH_RETRY)
> -		err = alua_rtpg(sdev, h->pg, 1);
> +		err = alua_rtpg(sdev, h->pg);
>  	kref_put(&h->pg->kref, release_port_group);
>  out:
>  	if (fn)

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> The current ALUA device_handler has two drawbacks:
> - We're sending a 'SET TARGET PORT GROUP' command to every LUN,
>   disregarding the fact that several LUNs might be in a port group
>   and will be automatically switched whenever _any_ LUN within
>   that port group receives the command.
> - Whenever a LUN is in 'transitioning' mode we cannot block I/O
>   to that LUN, instead the controller has to abort the command.
>   This leads to increased traffic across the wire and heavy load
>   on the controller during switchover.

I'm not sure I understand what this means - why couldn't we block I/O?
and what does 'heavy load' mean?  Aborting commands is 'heavy load'?

> 
> With this patch the RTPG handling is moved to a per-portgroup
> workqueue. This reduces the number of 'REPORT TARGET PORT GROUP'
> and 'SET TARGET PORT GROUPS' sent to the controller as we're sending
> them now per port group, and not per device as previously.
> It also allows us to block I/O to any LUN / port group found to be
> in 'transitioning' ALUA mode, as the workqueue item will be requeued
> until the controller moves out of transitioning.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 414 ++++++++++++++++++++++++-----
>  1 file changed, 349 insertions(+), 65 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index b52db8b..85fd597 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -22,6 +22,8 @@
>  #include <linux/slab.h>
>  #include <linux/delay.h>
>  #include <linux/module.h>
> +#include <linux/workqueue.h>
> +#include <linux/rcupdate.h>
>  #include <asm/unaligned.h>
>  #include <scsi/scsi.h>
>  #include <scsi/scsi_dbg.h>
> @@ -59,10 +61,15 @@
>  #define ALUA_RTPG_SIZE			128
>  #define ALUA_FAILOVER_TIMEOUT		60
>  #define ALUA_FAILOVER_RETRIES		5
> +#define ALUA_RTPG_DELAY_MSECS		5
>  
>  /* device handler flags */
> -#define ALUA_OPTIMIZE_STPG		1
> -#define ALUA_RTPG_EXT_HDR_UNSUPP	2
> +#define ALUA_OPTIMIZE_STPG		0x01
> +#define ALUA_RTPG_EXT_HDR_UNSUPP	0x02
> +/* State machine flags */
> +#define ALUA_PG_RUN_RTPG		0x10
> +#define ALUA_PG_RUN_STPG		0x20
> +
>  
>  static LIST_HEAD(port_group_list);
>  static DEFINE_SPINLOCK(port_group_lock);
> @@ -78,13 +85,28 @@ struct alua_port_group {
>  	int			pref;
>  	unsigned		flags; /* used for optimizing STPG */
>  	unsigned char		transition_tmo;
> +	unsigned long		expiry;
> +	unsigned long		interval;
> +	struct workqueue_struct *work_q;
> +	struct delayed_work	rtpg_work;
> +	struct delayed_work	stpg_work;
> +	struct delayed_work	qdata_work;
> +	spinlock_t		rtpg_lock;
> +	struct list_head	rtpg_list;
> +	struct scsi_device	*rtpg_sdev;
>  };
>  
>  struct alua_dh_data {
>  	struct alua_port_group	*pg;
> +	spinlock_t		pg_lock;
>  	int			rel_port;
>  	int			tpgs;
> -	struct scsi_device	*sdev;
> +	int			init_error;
> +	struct mutex		init_mutex;
> +};
> +
> +struct alua_queue_data {
> +	struct list_head	entry;
>  	activate_complete	callback_fn;
>  	void			*callback_data;
>  };
> @@ -93,6 +115,10 @@ struct alua_dh_data {
>  #define ALUA_POLICY_SWITCH_ALL		1
>  
>  static char print_alua_state(int);
> +static void alua_rtpg_work(struct work_struct *work);
> +static void alua_stpg_work(struct work_struct *work);
> +static void alua_qdata_work(struct work_struct *work);
> +static void alua_check(struct scsi_device *sdev);
>  
>  static void release_port_group(struct kref *kref)
>  {
> @@ -103,6 +129,9 @@ static void release_port_group(struct kref *kref)
>  	spin_lock(&port_group_lock);
>  	list_del(&pg->node);
>  	spin_unlock(&port_group_lock);
> +	WARN_ON(pg->rtpg_sdev);
> +	if (pg->work_q)
> +		destroy_workqueue(pg->work_q);
>  	kfree(pg);
>  }
>  
> @@ -296,13 +325,17 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		if (strncmp(tmp_pg->device_id_str, device_id_str,
>  			    device_id_size))
>  			continue;
> -		h->pg = tmp_pg;
>  		kref_get(&tmp_pg->kref);
> +		spin_lock(&h->pg_lock);
> +		rcu_assign_pointer(h->pg, tmp_pg);
> +		spin_unlock(&h->pg_lock);
>  		break;
>  	}
>  	spin_unlock(&port_group_lock);
> -	if (h->pg)
> +	if (h->pg) {
> +		synchronize_rcu();
>  		return SCSI_DH_OK;
> +	}
>  
>  	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
>  	if (!pg) {
> @@ -322,6 +355,19 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  	pg->tpgs = h->tpgs;
>  	pg->state = TPGS_STATE_OPTIMIZED;
>  	kref_init(&pg->kref);
> +	INIT_DELAYED_WORK(&pg->rtpg_work, alua_rtpg_work);
> +	INIT_DELAYED_WORK(&pg->stpg_work, alua_stpg_work);
> +	INIT_DELAYED_WORK(&pg->qdata_work, alua_qdata_work);
> +	INIT_LIST_HEAD(&pg->rtpg_list);
> +	INIT_LIST_HEAD(&pg->node);
> +	spin_lock_init(&pg->rtpg_lock);
> +	pg->work_q = alloc_ordered_workqueue("alua_wp_%s_%d", WQ_MEM_RECLAIM,
> +					     pg->device_id_str, pg->group_id);
> +	if (!pg->work_q) {
> +		kref_put(&pg->kref, release_port_group);
> +		/* Temporary failure, bypass */
> +		return SCSI_DH_DEV_TEMP_BUSY;
> +	}
>  	spin_lock(&port_group_lock);
>  	/*
>  	 * Re-check list again to catch
> @@ -335,15 +381,19 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>  		if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
>  			    device_id_size))
>  			continue;
> -		h->pg = tmp_pg;
>  		kref_get(&tmp_pg->kref);
> +		spin_lock(&h->pg_lock);
> +		rcu_assign_pointer(h->pg, tmp_pg);
> +		spin_unlock(&h->pg_lock);
>  		kfree(pg);
>  		pg = NULL;
>  		break;
>  	}
>  	if (pg) {
>  		list_add(&pg->node, &port_group_list);
> -		h->pg = pg;
> +		spin_lock(&h->pg_lock);
> +		rcu_assign_pointer(h->pg, pg);
> +		spin_unlock(&h->pg_lock);
>  	}
>  	spin_unlock(&port_group_lock);
>  
> @@ -377,11 +427,14 @@ static int alua_check_sense(struct scsi_device *sdev,
>  {
>  	switch (sense_hdr->sense_key) {
>  	case NOT_READY:
> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0a)
> +		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0a) {
>  			/*
>  			 * LUN Not Accessible - ALUA state transition
> +			 * Kickoff worker to update internal state.
>  			 */
> +			alua_check(sdev);
>  			return ADD_TO_MLQUEUE;
> +		}
>  		break;
>  	case UNIT_ATTENTION:
>  		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
> @@ -429,14 +482,15 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
>  	unsigned char *ucp, *buff;
>  	unsigned err, retval;
> -	unsigned long expiry, interval = 0;
>  	unsigned int tpg_desc_tbl_off;
>  	unsigned char orig_transition_tmo;
>  
> -	if (!pg->transition_tmo)
> -		expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
> -	else
> -		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
> +	if (!pg->expiry) {
> +		if (!pg->transition_tmo)
> +			pg->expiry = round_jiffies_up(jiffies + ALUA_FAILOVER_TIMEOUT * HZ);
> +		else
> +			pg->expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
> +	}
>  
>  	buff = kzalloc(bufflen, GFP_KERNEL);
>  	if (!buff)
> @@ -455,6 +509,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  			else
>  				err = SCSI_DH_IO;
>  			kfree(buff);
> +			pg->expiry = 0;
>  			return err;
>  		}
>  
> @@ -481,16 +536,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  			err = SCSI_DH_RETRY;
>  		else if (sense_hdr.sense_key == UNIT_ATTENTION)
>  			err = SCSI_DH_RETRY;
> -		if (err == SCSI_DH_RETRY && time_before(jiffies, expiry)) {
> +		if (err == SCSI_DH_RETRY &&
> +		    pg->expiry != 0 && time_before(jiffies, pg->expiry)) {
>  			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
>  				    ALUA_DH_NAME);
>  			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> -			goto retry;
> +			return err;
>  		}
>  		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
>  			    ALUA_DH_NAME);
>  		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>  		kfree(buff);
> +		pg->expiry = 0;
>  		return SCSI_DH_IO;
>  	}
>  
> @@ -505,6 +562,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  			sdev_printk(KERN_WARNING, sdev,
>  				    "%s: kmalloc buffer failed\n",__func__);
>  			/* Temporary failure, bypass */
> +			pg->expiry = 0;
>  			return SCSI_DH_DEV_TEMP_BUSY;
>  		}
>  		goto retry;
> @@ -520,7 +578,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  		sdev_printk(KERN_INFO, sdev,
>  			    "%s: transition timeout set to %d seconds\n",
>  			    ALUA_DH_NAME, pg->transition_tmo);
> -		expiry = jiffies + pg->transition_tmo * HZ;
> +		pg->expiry = jiffies + pg->transition_tmo * HZ;
>  	}
>  
>  	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
> @@ -554,23 +612,26 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  
>  	switch (pg->state) {
>  	case TPGS_STATE_TRANSITIONING:
> -		if (time_before(jiffies, expiry)) {
> +		if (time_before(jiffies, pg->expiry)) {
>  			/* State transition, retry */
> -			interval += 2000;
> -			msleep(interval);
> -			goto retry;
> +			pg->interval = 2;
> +			err = SCSI_DH_RETRY;
> +		} else {
> +			/* Transitioning time exceeded, set port to standby */
> +			err = SCSI_DH_IO;
> +			pg->state = TPGS_STATE_STANDBY;
> +			pg->expiry = 0;
>  		}
> -		/* Transitioning time exceeded, set port to standby */
> -		err = SCSI_DH_RETRY;
> -		pg->state = TPGS_STATE_STANDBY;
>  		break;
>  	case TPGS_STATE_OFFLINE:
>  		/* Path unusable */
>  		err = SCSI_DH_DEV_OFFLINED;
> +		pg->expiry = 0;
>  		break;
>  	default:
>  		/* Useable path if active */
>  		err = SCSI_DH_OK;
> +		pg->expiry = 0;
>  		break;
>  	}
>  	kfree(buff);
> @@ -590,8 +651,8 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  	struct scsi_sense_hdr sense_hdr;
>  
>  	if (!(pg->tpgs & TPGS_MODE_EXPLICIT)) {
> -		/* Only implicit ALUA supported, retry */
> -		return SCSI_DH_RETRY;
> +		/* Only implicit ALUA supported */
> +		return SCSI_DH_OK;
>  	}
>  	switch (pg->state) {
>  	case TPGS_STATE_OPTIMIZED:
> @@ -617,8 +678,6 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  		return SCSI_DH_NOSYS;
>  		break;
>  	}
> -	/* Set state to transitioning */
> -	pg->state = TPGS_STATE_TRANSITIONING;
>  	retval = submit_stpg(sdev, pg->group_id, &sense_hdr);
>  
>  	if (retval) {
> @@ -638,6 +697,150 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  	return SCSI_DH_RETRY;
>  }
>  
> +static void alua_rtpg_work(struct work_struct *work)
> +{
> +	struct alua_port_group *pg =
> +		container_of(work, struct alua_port_group, rtpg_work.work);
> +	struct scsi_device *sdev;
> +	int err = SCSI_DH_OK;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&pg->rtpg_lock, flags);
> +	sdev = pg->rtpg_sdev;
> +	if (!sdev) {
> +		WARN_ON(pg->flags & ALUA_PG_RUN_RTPG ||
> +			pg->flags & ALUA_PG_RUN_STPG);
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		return;
> +	}
> +	if (pg->flags & ALUA_PG_RUN_RTPG) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		err = alua_rtpg(sdev, pg);
> +		if (err == SCSI_DH_RETRY) {
> +			queue_delayed_work(pg->work_q, &pg->rtpg_work,
> +					   pg->interval * HZ);
> +			return;
> +		}
> +		spin_lock_irqsave(&pg->rtpg_lock, flags);
> +		pg->flags &= ~ALUA_PG_RUN_RTPG;
> +		if (err != SCSI_DH_OK)
> +			pg->flags &= ~ALUA_PG_RUN_STPG;
> +	}
> +	if (pg->flags & ALUA_PG_RUN_STPG) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		queue_delayed_work(pg->work_q, &pg->stpg_work, 0);
> +		return;
> +	}
> +	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +	queue_delayed_work(pg->work_q, &pg->qdata_work, 0);
> +}
> +
> +static void alua_stpg_work(struct work_struct *work)
> +{
> +	struct alua_port_group *pg =
> +		container_of(work, struct alua_port_group, stpg_work.work);
> +	struct scsi_device *sdev;
> +	int err = SCSI_DH_OK;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&pg->rtpg_lock, flags);
> +	sdev = pg->rtpg_sdev;
> +	if (!sdev) {
> +		WARN_ON(pg->flags & ALUA_PG_RUN_STPG);
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		return;
> +	}
> +
> +	if (pg->flags & ALUA_PG_RUN_STPG) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		err = alua_stpg(sdev, pg);
> +		spin_lock_irqsave(&pg->rtpg_lock, flags);
> +		pg->flags &= ~ALUA_PG_RUN_STPG;
> +		if (err == SCSI_DH_RETRY) {
> +			pg->flags |= ALUA_PG_RUN_RTPG;
> +			pg->interval = 0;
> +		}
> +	}
> +	if (pg->flags & ALUA_PG_RUN_RTPG) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		queue_delayed_work(pg->work_q, &pg->rtpg_work,
> +				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
> +		return;
> +	}
> +
> +	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +	queue_delayed_work(pg->work_q, &pg->qdata_work, 0);
> +}
> +
> +static void alua_qdata_work(struct work_struct *work)
> +{
> +	struct alua_port_group *pg =
> +		container_of(work, struct alua_port_group, qdata_work.work);
> +	struct scsi_device *sdev;
> +	LIST_HEAD(qdata_list);
> +	int err = SCSI_DH_OK;
> +	struct alua_queue_data *qdata, *tmp;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&pg->rtpg_lock, flags);
> +	sdev = pg->rtpg_sdev;
> +	if (WARN_ON(!sdev)) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		return;
> +	}
> +	if (pg->flags & ALUA_PG_RUN_RTPG) {
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		queue_delayed_work(pg->work_q, &pg->rtpg_work,
> +				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
> +		return;
> +	}
> +
> +	list_splice_init(&pg->rtpg_list, &qdata_list);
> +	pg->rtpg_sdev = NULL;
> +	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +
> +	list_for_each_entry_safe(qdata, tmp, &qdata_list, entry) {
> +		list_del(&qdata->entry);
> +		if (qdata->callback_fn)
> +			qdata->callback_fn(qdata->callback_data, err);
> +		kfree(qdata);
> +	}
> +	kref_put(&pg->kref, release_port_group);
> +	scsi_device_put(sdev);
> +}
> +
> +static void alua_rtpg_queue(struct alua_port_group *pg,
> +			    struct scsi_device *sdev,
> +			    struct alua_queue_data *qdata)
> +{
> +	int start_queue = 0;
> +	unsigned long flags;
> +
> +	if (!pg)
> +		return;
> +
> +	kref_get(&pg->kref);
> +	spin_lock_irqsave(&pg->rtpg_lock, flags);
> +	if (qdata) {
> +		list_add_tail(&qdata->entry, &pg->rtpg_list);
> +		pg->flags |= ALUA_PG_RUN_STPG;
> +	}
> +	if (pg->rtpg_sdev == NULL) {
> +		pg->interval = 0;
> +		pg->flags |= ALUA_PG_RUN_RTPG;
> +		kref_get(&pg->kref);
> +		pg->rtpg_sdev = sdev;
> +		scsi_device_get(sdev);
> +		start_queue = 1;
> +	}
> +	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +
> +	if (start_queue)
> +		queue_delayed_work(pg->work_q, &pg->rtpg_work,
> +				   msecs_to_jiffies(ALUA_RTPG_DELAY_MSECS));
> +	kref_put(&pg->kref, release_port_group);
> +}
> +
>  /*
>   * alua_initialize - Initialize ALUA state
>   * @sdev: the device to be initialized
> @@ -647,22 +850,35 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)
>   */
>  static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  {
> -	int err;
> -
> -	err = alua_check_tpgs(sdev, h);
> -	if (err != SCSI_DH_OK)
> -		goto out;
> -
> -	err = alua_check_vpd(sdev, h);
> -	if (err != SCSI_DH_OK || !h->pg)
> -		goto out;
> -
> -	kref_get(&h->pg->kref);
> -	err = alua_rtpg(sdev, h->pg);
> -	kref_put(&h->pg->kref, release_port_group);
> -out:
> -	return err;
> +	struct alua_port_group *pg = NULL;
> +	int error;
> +
> +	mutex_lock(&h->init_mutex);
> +	error = alua_check_tpgs(sdev, h);
> +	if (error == SCSI_DH_OK) {
> +		error = alua_check_vpd(sdev, h);
> +		rcu_read_lock();
> +		pg = rcu_dereference(h->pg);
> +		if (!pg) {
> +			rcu_read_unlock();
> +			h->tpgs = TPGS_MODE_NONE;
> +			error = SCSI_DH_DEV_UNSUPP;
> +		} else {
> +			WARN_ON(error != SCSI_DH_OK);
> +			kref_get(&pg->kref);
> +			rcu_read_unlock();
> +		}
> +	}
> +	h->init_error = error;
> +	mutex_unlock(&h->init_mutex);
> +	if (pg) {
> +		pg->expiry = 0;
> +		alua_rtpg_queue(pg, sdev, NULL);
> +		kref_put(&pg->kref, release_port_group);
> +	}
> +	return error;
>  }
> +
>  /*
>   * alua_set_params - set/unset the optimize flag
>   * @sdev: device on the path to be activated
> @@ -679,6 +895,10 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
>  	unsigned int optimize = 0, argc;
>  	const char *p = params;
>  	int result = SCSI_DH_OK;
> +	unsigned long flags;
> +
> +	if (!h)
> +		return -ENXIO;
>  
>  	if ((sscanf(params, "%u", &argc) != 1) || (argc != 1))
>  		return -EINVAL;
> @@ -694,11 +914,12 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)
>  		rcu_read_unlock();
>  		return -ENXIO;
>  	}
> -
> +	spin_lock_irqsave(&pg->rtpg_lock, flags);
>  	if (optimize)
>  		pg->flags |= ALUA_OPTIMIZE_STPG;
>  	else
>  		pg->flags |= ~ALUA_OPTIMIZE_STPG;
> +	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>  	rcu_read_unlock();
>  
>  	return result;
> @@ -723,24 +944,46 @@ static int alua_activate(struct scsi_device *sdev,
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
>  	int err = SCSI_DH_OK;
> +	struct alua_queue_data *qdata;
> +	struct alua_port_group *pg;
>  
> -	if (!h->pg)
> +	if (!h) {
> +		err = SCSI_DH_NOSYS;
>  		goto out;
> +	}
>  
> -	kref_get(&h->pg->kref);
> -
> -	if (optimize_stpg)
> -		h->pg->flags |= ALUA_OPTIMIZE_STPG;
> +	qdata = kzalloc(sizeof(*qdata), GFP_KERNEL);
> +	if (!qdata) {
> +		err = SCSI_DH_RES_TEMP_UNAVAIL;
> +		goto out;
> +	}
> +	qdata->callback_fn = fn;
> +	qdata->callback_data = data;
>  
> -	err = alua_rtpg(sdev, h->pg);
> -	if (err != SCSI_DH_OK) {
> -		kref_put(&h->pg->kref, release_port_group);
> +	mutex_lock(&h->init_mutex);
> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (!pg) {
> +		rcu_read_unlock();
> +		kfree(qdata);
> +		err = h->init_error;
> +		mutex_unlock(&h->init_mutex);
>  		goto out;
>  	}
> -	err = alua_stpg(sdev, h->pg);
> -	if (err == SCSI_DH_RETRY)
> -		err = alua_rtpg(sdev, h->pg);
> -	kref_put(&h->pg->kref, release_port_group);
> +	mutex_unlock(&h->init_mutex);
> +	fn = NULL;
> +	kref_get(&pg->kref);
> +	rcu_read_unlock();
> +
> +	if (optimize_stpg) {
> +		unsigned long flags;
> +
> +		spin_lock_irqsave(&pg->rtpg_lock, flags);
> +		pg->flags |= ALUA_OPTIMIZE_STPG;
> +		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +	}
> +	alua_rtpg_queue(pg, sdev, qdata);
> +	kref_put(&pg->kref, release_port_group);
>  out:
>  	if (fn)
>  		fn(data, err);
> @@ -748,6 +991,30 @@ out:
>  }
>  
>  /*
> + * alua_check - check path status
> + * @sdev: device on the path to be checked
> + *
> + * Check the device status
> + */
> +static void alua_check(struct scsi_device *sdev)
> +{
> +	struct alua_dh_data *h = sdev->handler_data;
> +	struct alua_port_group *pg;
> +
> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (!pg) {
> +		rcu_read_unlock();
> +		return;
> +	}
> +	kref_get(&pg->kref);
> +	rcu_read_unlock();
> +	alua_rtpg_queue(pg, sdev, NULL);
> +	kref_put(&pg->kref, release_port_group);
> +	rcu_read_unlock();
> +}
> +
> +/*
>   * alua_prep_fn - request callback
>   *
>   * Fail I/O to all paths not in state
> @@ -756,14 +1023,22 @@ out:
>  static int alua_prep_fn(struct scsi_device *sdev, struct request *req)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
> -	int state;
> +	struct alua_port_group *pg;
> +	int state = TPGS_STATE_OPTIMIZED;
>  	int ret = BLKPREP_OK;
>  
> -	if (!h->pg)
> +	if (!h)
>  		return ret;
> -	kref_get(&h->pg->kref);
> -	state = h->pg->state;
> -	kref_put(&h->pg->kref, release_port_group);
> +
> +	rcu_read_lock();
> +	pg = rcu_dereference(h->pg);
> +	if (pg) {
> +		state = pg->state;
> +		/* Defer I/O while rtpg_work is active */
> +		if (pg->rtpg_sdev)
> +			state = TPGS_STATE_TRANSITIONING;
> +	}
> +	rcu_read_unlock();
>  	if (state == TPGS_STATE_TRANSITIONING)
>  		ret = BLKPREP_DEFER;
>  	else if (state != TPGS_STATE_OPTIMIZED &&
> @@ -788,11 +1063,13 @@ static int alua_bus_attach(struct scsi_device *sdev)
>  	h = kzalloc(sizeof(*h) , GFP_KERNEL);
>  	if (!h)
>  		return -ENOMEM;
> +	spin_lock_init(&h->pg_lock);
>  	h->tpgs = TPGS_MODE_UNINITIALIZED;
> -	h->pg = NULL;
> +	rcu_assign_pointer(h->pg, NULL);
>  	h->rel_port = -1;
> -	h->sdev = sdev;
> +	h->init_error = SCSI_DH_OK;
>  
> +	mutex_init(&h->init_mutex);
>  	err = alua_initialize(sdev, h);
>  	if (err != SCSI_DH_OK && err != SCSI_DH_DEV_OFFLINED)
>  		goto failed;
> @@ -811,10 +1088,17 @@ failed:
>  static void alua_bus_detach(struct scsi_device *sdev)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
> +	struct alua_port_group *pg;
>  
> -	if (h->pg) {
> -		kref_put(&h->pg->kref, release_port_group);
> -		h->pg = NULL;
> +	spin_lock(&h->pg_lock);
> +	pg = h->pg;
> +	rcu_assign_pointer(h->pg, NULL);
> +	spin_unlock(&h->pg_lock);
> +	synchronize_rcu();
> +	if (pg) {
> +		if (pg->rtpg_sdev)
> +			flush_workqueue(pg->work_q);
> +		kref_put(&pg->kref, release_port_group);
>  	}
>  	sdev->handler_data = NULL;
>  	kfree(h);

So, you've already had a bit of discussion with Christoph about this,
the main portion of your ALUA rewrite, and I won't go over all of that,
except to say that I'd have to agree that having separate work queues
for the different RTPG/STPG functions and having them manipulate each
other's flags seems like we'd be better off having just one work
function that did everything.  Less messy and easier to maintain.

Also, it seems like wrong ordering of kref_get() vs. scsi_device_get(),
in alua_rtpg_queue() since they are released as kref_put() then
scsi_device_put()?

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 20/23] scsi_dh_alua: Recheck state on unit attention

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> When we receive a unit attention code of 'ALUA state changed'
> we should recheck the state, as it might be due to an implicit
> ALUA state transition.
> At the same time a workqueue item might already be queued, which
> should be started immediately to avoid any delays.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 48 +++++++++++++++---------------
>  1 file changed, 24 insertions(+), 24 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 85fd597..8717fd3 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -118,7 +118,7 @@ static char print_alua_state(int);
>  static void alua_rtpg_work(struct work_struct *work);
>  static void alua_stpg_work(struct work_struct *work);
>  static void alua_qdata_work(struct work_struct *work);
> -static void alua_check(struct scsi_device *sdev);
> +static void alua_check(struct scsi_device *sdev, bool force);
>  
>  static void release_port_group(struct kref *kref)
>  {
> @@ -423,7 +423,7 @@ static char print_alua_state(int state)
>  }
>  
>  static int alua_check_sense(struct scsi_device *sdev,
> -			    struct scsi_sense_hdr *sense_hdr)
> +			     struct scsi_sense_hdr *sense_hdr)
>  {
>  	switch (sense_hdr->sense_key) {
>  	case NOT_READY:
> @@ -432,36 +432,34 @@ static int alua_check_sense(struct scsi_device *sdev,
>  			 * LUN Not Accessible - ALUA state transition
>  			 * Kickoff worker to update internal state.
>  			 */
> -			alua_check(sdev);
> -			return ADD_TO_MLQUEUE;
> +			alua_check(sdev, false);
> +			return NEEDS_RETRY;
>  		}
>  		break;
>  	case UNIT_ATTENTION:
> -		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
> -			/*
> -			 * Power On, Reset, or Bus Device Reset, just retry.
> -			 */
> -			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x04)
> -			/*
> -			 * Device internal reset
> -			 */
> -			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
> +		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) {
>  			/*
> -			 * Mode Parameter Changed
> +			 * Power On, Reset, or Bus Device Reset.
> +			 * Might have obscured a state transition,
> +			 * so schedule a recheck.
>  			 */
> +			alua_check(sdev, true);
>  			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
> +		}
> +		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06) {
>  			/*
>  			 * ALUA state changed
>  			 */
> +			alua_check(sdev, true);
>  			return ADD_TO_MLQUEUE;
> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07)
> +		}
> +		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07) {
>  			/*
>  			 * Implicit ALUA state transition failed
>  			 */
> +			alua_check(sdev, true);
>  			return ADD_TO_MLQUEUE;
> +		}
>  		break;
>  	}
>  
> @@ -811,7 +809,7 @@ static void alua_qdata_work(struct work_struct *work)
>  
>  static void alua_rtpg_queue(struct alua_port_group *pg,
>  			    struct scsi_device *sdev,
> -			    struct alua_queue_data *qdata)
> +			    struct alua_queue_data *qdata, bool force)
>  {
>  	int start_queue = 0;
>  	unsigned long flags;
> @@ -832,7 +830,9 @@ static void alua_rtpg_queue(struct alua_port_group *pg,
>  		pg->rtpg_sdev = sdev;
>  		scsi_device_get(sdev);
>  		start_queue = 1;
> -	}
> +	} else if (!(pg->flags & ALUA_PG_RUN_RTPG) && force)
> +		start_queue = 1;
> +
>  	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>  
>  	if (start_queue)
> @@ -873,7 +873,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>  	mutex_unlock(&h->init_mutex);
>  	if (pg) {
>  		pg->expiry = 0;
> -		alua_rtpg_queue(pg, sdev, NULL);
> +		alua_rtpg_queue(pg, sdev, NULL, true);
>  		kref_put(&pg->kref, release_port_group);
>  	}
>  	return error;
> @@ -982,7 +982,7 @@ static int alua_activate(struct scsi_device *sdev,
>  		pg->flags |= ALUA_OPTIMIZE_STPG;
>  		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>  	}
> -	alua_rtpg_queue(pg, sdev, qdata);
> +	alua_rtpg_queue(pg, sdev, qdata, true);
>  	kref_put(&pg->kref, release_port_group);
>  out:
>  	if (fn)
> @@ -996,7 +996,7 @@ out:
>   *
>   * Check the device status
>   */
> -static void alua_check(struct scsi_device *sdev)
> +static void alua_check(struct scsi_device *sdev, bool force)
>  {
>  	struct alua_dh_data *h = sdev->handler_data;
>  	struct alua_port_group *pg;
> @@ -1009,7 +1009,7 @@ static void alua_check(struct scsi_device *sdev)
>  	}
>  	kref_get(&pg->kref);
>  	rcu_read_unlock();
> -	alua_rtpg_queue(pg, sdev, NULL);
> +	alua_rtpg_queue(pg, sdev, NULL, force);
>  	kref_put(&pg->kref, release_port_group);
>  	rcu_read_unlock();
>  }

I think this change goes along with the previous patch in terms of the
design, so we'll have to see how that turns out.  Fundamentally, though,
this changes the handling for LUN Not Accessible - ALUA state transition
to be NEEDS_RETRY instead of ADD_TO_MLQUEUE, so what is the effect of that?

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 21/23] scsi_dh_alua: update all port states

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> When we read in the target port group state we should be
> updating all affected port groups, otherwise we risk
> running out of sync.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 21 +++++++++++++++++----
>  1 file changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 8717fd3..2fa985e 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -477,11 +477,13 @@ static int alua_check_sense(struct scsi_device *sdev,
>  static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  {
>  	struct scsi_sense_hdr sense_hdr;
> +	struct alua_port_group *tmp_pg;
>  	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
>  	unsigned char *ucp, *buff;
>  	unsigned err, retval;
>  	unsigned int tpg_desc_tbl_off;
>  	unsigned char orig_transition_tmo;
> +	unsigned long flags;
>  
>  	if (!pg->expiry) {
>  		if (!pg->transition_tmo)
> @@ -584,17 +586,28 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>  	else
>  		tpg_desc_tbl_off = 4;
>  
> +	spin_lock_irqsave(&port_group_lock, flags);
>  	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
>  	     k < len;
>  	     k += off, ucp += off) {

"ucp" is not really a good variable name, it matches nothing in the T10
spec.  The structure is a "Target Port Group Descriptor".  I realize it
was already there, but maybe this is a good opportunity to change it?

>  
> -		if (pg->group_id == get_unaligned_be16(&ucp[2])) {
> -			pg->state = ucp[0] & 0x0f;
> -			pg->pref = ucp[0] >> 7;
> -			valid_states = ucp[1];
> +		list_for_each_entry(tmp_pg, &port_group_list, node) {
> +			u16 group_id = get_unaligned_be16(&ucp[2]);

This is invariant and should be outside the loop.

> +			if (tmp_pg->group_id != group_id)
> +				continue;
> +			if (tmp_pg->device_id_size != pg->device_id_size)
> +				continue;
> +			if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
> +				    tmp_pg->device_id_size))
> +				continue;
> +			tmp_pg->state = ucp[0] & 0x0f;
> +			tmp_pg->pref = ucp[0] >> 7;
> +			if (tmp_pg == pg)
> +				valid_states = ucp[1];
>  		}
>  		off = 8 + (ucp[7] * 4);
>  	}
> +	spin_unlock_irqrestore(&port_group_lock, flags);
>  
>  	sdev_printk(KERN_INFO, sdev,
>  		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 22/23] scsi_dh_alua: Send TEST UNIT READY to poll for transitioning

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Sending a 'REPORT TARGET PORT GROUP' command is a costly operation,
> as the array has to gather information about all ports.
> So instead of using RTPG to poll for a status update when a port
> is in transitioning we should be sending a TEST UNIT READY, and
> wait for the sense code to report success.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 33 ++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 2fa985e..3c6b365 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -467,6 +467,30 @@ static int alua_check_sense(struct scsi_device *sdev,
>  }
>  
>  /*
> + * alua_tur - Send a TEST UNIT READY
> + * @sdev: device to which the TEST UNIT READY command should be send
> + *
> + * Send a TEST UNIT READY to @sdev to figure out the device state
> + * Returns SCSI_DH_RETRY if the sense code is NOT READY/ALUA TRANSITIONING,
> + * SCSI_DH_OK if no error occured, and SCSI_DH_IO otherwise.
> + */
> +static int alua_tur(struct scsi_device *sdev)
> +{
> +	struct scsi_sense_hdr sense_hdr;
> +	int retval;
> +
> +	retval = scsi_test_unit_ready(sdev, ALUA_FAILOVER_TIMEOUT * HZ,
> +				      ALUA_FAILOVER_RETRIES, &sense_hdr);
> +	if (sense_hdr.sense_key == NOT_READY &&
> +	    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
> +		return SCSI_DH_RETRY;
> +	else if (retval)
> +		return SCSI_DH_IO;
> +	else
> +		return SCSI_DH_OK;
> +}
> +
> +/*
>   * alua_rtpg - Evaluate REPORT TARGET GROUP STATES
>   * @sdev: the device to be evaluated.
>   *
> @@ -725,7 +749,16 @@ static void alua_rtpg_work(struct work_struct *work)
>  		return;
>  	}
>  	if (pg->flags & ALUA_PG_RUN_RTPG) {
> +		int state = pg->state;
>  		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
> +		if (state == TPGS_STATE_TRANSITIONING) {
> +			if (alua_tur(sdev) == SCSI_DH_RETRY) {
> +				queue_delayed_work(pg->work_q, &pg->rtpg_work,
> +						   pg->interval * HZ);
> +				return;
> +			}
> +			/* Send RTPG on failure or if TUR indicates SUCCESS */
> +		}
>  		err = alua_rtpg(sdev, pg);
>  		if (err == SCSI_DH_RETRY) {
>  			queue_delayed_work(pg->work_q, &pg->rtpg_work,

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 23/23] scsi_dh_alua: Update version to 2.0

[Ewan Milne]

On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/scsi/device_handler/scsi_dh_alua.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
> index 3c6b365..5b15936 100644
> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
> @@ -31,7 +31,7 @@
>  #include <scsi/scsi_dh.h>
>  
>  #define ALUA_DH_NAME "alua"
> -#define ALUA_DH_VER "1.3"
> +#define ALUA_DH_VER "2.0"
>  
>  #define TPGS_STATE_OPTIMIZED		0x0
>  #define TPGS_STATE_NONOPTIMIZED		0x1

Reviewed-by: Ewan D. Milne <emilne@redhat.com>

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Hannes Reinecke]

On 09/22/2015 09:49 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> The current ALUA device_handler has two drawbacks:
>> - We're sending a 'SET TARGET PORT GROUP' command to every LUN,
>>   disregarding the fact that several LUNs might be in a port group
>>   and will be automatically switched whenever _any_ LUN within
>>   that port group receives the command.
>> - Whenever a LUN is in 'transitioning' mode we cannot block I/O
>>   to that LUN, instead the controller has to abort the command.
>>   This leads to increased traffic across the wire and heavy load
>>   on the controller during switchover.
> 
> I'm not sure I understand what this means - why couldn't we block I/O?
> and what does 'heavy load' mean?  Aborting commands is 'heavy load'?
> 
If we're getting a sense code indicating that the LUN is in
transitioning _and_ we're blocking I/O we never ever send down I/Os to
that driver anymore, so we cannot receive any sense codes indicating the
transitioning is done.
At the same time, every I/O we're sending down will be returned by the
storage I/O with a sense code, requiring us to retry the command.
Hence we're constantly retrying I/O.

[ .. ]
>> @@ -811,10 +1088,17 @@ failed:
>>  static void alua_bus_detach(struct scsi_device *sdev)
>>  {
>>  	struct alua_dh_data *h = sdev->handler_data;
>> +	struct alua_port_group *pg;
>>  
>> -	if (h->pg) {
>> -		kref_put(&h->pg->kref, release_port_group);
>> -		h->pg = NULL;
>> +	spin_lock(&h->pg_lock);
>> +	pg = h->pg;
>> +	rcu_assign_pointer(h->pg, NULL);
>> +	spin_unlock(&h->pg_lock);
>> +	synchronize_rcu();
>> +	if (pg) {
>> +		if (pg->rtpg_sdev)
>> +			flush_workqueue(pg->work_q);
>> +		kref_put(&pg->kref, release_port_group);
>>  	}
>>  	sdev->handler_data = NULL;
>>  	kfree(h);
> 
> So, you've already had a bit of discussion with Christoph about this,
> the main portion of your ALUA rewrite, and I won't go over all of that,
> except to say that I'd have to agree that having separate work queues
> for the different RTPG/STPG functions and having them manipulate each
> other's flags seems like we'd be better off having just one work
> function that did everything.  Less messy and easier to maintain.
> 
> Also, it seems like wrong ordering of kref_get() vs. scsi_device_get(),
> in alua_rtpg_queue() since they are released as kref_put() then
> scsi_device_put()?
> 
Yeah, I've reworked the reference counting.
And reverted the workqueue handling to use the original model.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 21/23] scsi_dh_alua: update all port states

[Hannes Reinecke]

On 09/22/2015 10:04 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> When we read in the target port group state we should be
>> updating all affected port groups, otherwise we risk
>> running out of sync.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 21 +++++++++++++++++----
>>  1 file changed, 17 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index 8717fd3..2fa985e 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -477,11 +477,13 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>>  {
>>  	struct scsi_sense_hdr sense_hdr;
>> +	struct alua_port_group *tmp_pg;
>>  	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
>>  	unsigned char *ucp, *buff;
>>  	unsigned err, retval;
>>  	unsigned int tpg_desc_tbl_off;
>>  	unsigned char orig_transition_tmo;
>> +	unsigned long flags;
>>  
>>  	if (!pg->expiry) {
>>  		if (!pg->transition_tmo)
>> @@ -584,17 +586,28 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
>>  	else
>>  		tpg_desc_tbl_off = 4;
>>  
>> +	spin_lock_irqsave(&port_group_lock, flags);
>>  	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
>>  	     k < len;
>>  	     k += off, ucp += off) {
> 
> "ucp" is not really a good variable name, it matches nothing in the T10
> spec.  The structure is a "Target Port Group Descriptor".  I realize it
> was already there, but maybe this is a good opportunity to change it?
> 
Doug Gilbert is to blame; I've copied this section over from sg3_utils :-)
But yeah, I can set it to eg 'desc'.

>>  
>> -		if (pg->group_id == get_unaligned_be16(&ucp[2])) {
>> -			pg->state = ucp[0] & 0x0f;
>> -			pg->pref = ucp[0] >> 7;
>> -			valid_states = ucp[1];
>> +		list_for_each_entry(tmp_pg, &port_group_list, node) {
>> +			u16 group_id = get_unaligned_be16(&ucp[2]);
> 
> This is invariant and should be outside the loop.
> 
True.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 20/23] scsi_dh_alua: Recheck state on unit attention

[Hannes Reinecke]

On 09/22/2015 09:57 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> When we receive a unit attention code of 'ALUA state changed'
>> we should recheck the state, as it might be due to an implicit
>> ALUA state transition.
>> At the same time a workqueue item might already be queued, which
>> should be started immediately to avoid any delays.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 48 +++++++++++++++---------------
>>  1 file changed, 24 insertions(+), 24 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index 85fd597..8717fd3 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -118,7 +118,7 @@ static char print_alua_state(int);
>>  static void alua_rtpg_work(struct work_struct *work);
>>  static void alua_stpg_work(struct work_struct *work);
>>  static void alua_qdata_work(struct work_struct *work);
>> -static void alua_check(struct scsi_device *sdev);
>> +static void alua_check(struct scsi_device *sdev, bool force);
>>  
>>  static void release_port_group(struct kref *kref)
>>  {
>> @@ -423,7 +423,7 @@ static char print_alua_state(int state)
>>  }
>>  
>>  static int alua_check_sense(struct scsi_device *sdev,
>> -			    struct scsi_sense_hdr *sense_hdr)
>> +			     struct scsi_sense_hdr *sense_hdr)
>>  {
>>  	switch (sense_hdr->sense_key) {
>>  	case NOT_READY:
>> @@ -432,36 +432,34 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  			 * LUN Not Accessible - ALUA state transition
>>  			 * Kickoff worker to update internal state.
>>  			 */
>> -			alua_check(sdev);
>> -			return ADD_TO_MLQUEUE;
>> +			alua_check(sdev, false);
>> +			return NEEDS_RETRY;
>>  		}
>>  		break;
>>  	case UNIT_ATTENTION:
>> -		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
>> -			/*
>> -			 * Power On, Reset, or Bus Device Reset, just retry.
>> -			 */
>> -			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x04)
>> -			/*
>> -			 * Device internal reset
>> -			 */
>> -			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
>> +		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) {
>>  			/*
>> -			 * Mode Parameter Changed
>> +			 * Power On, Reset, or Bus Device Reset.
>> +			 * Might have obscured a state transition,
>> +			 * so schedule a recheck.
>>  			 */
>> +			alua_check(sdev, true);
>>  			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
>> +		}
>> +		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06) {
>>  			/*
>>  			 * ALUA state changed
>>  			 */
>> +			alua_check(sdev, true);
>>  			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07)
>> +		}
>> +		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x07) {
>>  			/*
>>  			 * Implicit ALUA state transition failed
>>  			 */
>> +			alua_check(sdev, true);
>>  			return ADD_TO_MLQUEUE;
>> +		}
>>  		break;
>>  	}
>>  
>> @@ -811,7 +809,7 @@ static void alua_qdata_work(struct work_struct *work)
>>  
>>  static void alua_rtpg_queue(struct alua_port_group *pg,
>>  			    struct scsi_device *sdev,
>> -			    struct alua_queue_data *qdata)
>> +			    struct alua_queue_data *qdata, bool force)
>>  {
>>  	int start_queue = 0;
>>  	unsigned long flags;
>> @@ -832,7 +830,9 @@ static void alua_rtpg_queue(struct alua_port_group *pg,
>>  		pg->rtpg_sdev = sdev;
>>  		scsi_device_get(sdev);
>>  		start_queue = 1;
>> -	}
>> +	} else if (!(pg->flags & ALUA_PG_RUN_RTPG) && force)
>> +		start_queue = 1;
>> +
>>  	spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>>  
>>  	if (start_queue)
>> @@ -873,7 +873,7 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)
>>  	mutex_unlock(&h->init_mutex);
>>  	if (pg) {
>>  		pg->expiry = 0;
>> -		alua_rtpg_queue(pg, sdev, NULL);
>> +		alua_rtpg_queue(pg, sdev, NULL, true);
>>  		kref_put(&pg->kref, release_port_group);
>>  	}
>>  	return error;
>> @@ -982,7 +982,7 @@ static int alua_activate(struct scsi_device *sdev,
>>  		pg->flags |= ALUA_OPTIMIZE_STPG;
>>  		spin_unlock_irqrestore(&pg->rtpg_lock, flags);
>>  	}
>> -	alua_rtpg_queue(pg, sdev, qdata);
>> +	alua_rtpg_queue(pg, sdev, qdata, true);
>>  	kref_put(&pg->kref, release_port_group);
>>  out:
>>  	if (fn)
>> @@ -996,7 +996,7 @@ out:
>>   *
>>   * Check the device status
>>   */
>> -static void alua_check(struct scsi_device *sdev)
>> +static void alua_check(struct scsi_device *sdev, bool force)
>>  {
>>  	struct alua_dh_data *h = sdev->handler_data;
>>  	struct alua_port_group *pg;
>> @@ -1009,7 +1009,7 @@ static void alua_check(struct scsi_device *sdev)
>>  	}
>>  	kref_get(&pg->kref);
>>  	rcu_read_unlock();
>> -	alua_rtpg_queue(pg, sdev, NULL);
>> +	alua_rtpg_queue(pg, sdev, NULL, force);
>>  	kref_put(&pg->kref, release_port_group);
>>  	rcu_read_unlock();
>>  }
> 
> I think this change goes along with the previous patch in terms of the
> design, so we'll have to see how that turns out.  Fundamentally, though,
> this changes the handling for LUN Not Accessible - ALUA state transition
> to be NEEDS_RETRY instead of ADD_TO_MLQUEUE, so what is the effect of that?
> 
ADD_TO_MLQUEUE will always be retried, NEEDS_RETRY is bounded by the
number of retries.
Originally we had to return ADD_TO_MLQUEUE so to be sure to retry until
the transition is done, causing quite some retries.
As now we're polling the device from a workqueue we can terminate
the number of retries by returning NEEDS_RETRY, knowing that the polling
(and/or sense code evaluation) will tell us when the transition is finished.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Ewan Milne]

On Tue, 2015-09-22 at 22:15 +0200, Hannes Reinecke wrote:
> On 09/22/2015 09:49 PM, Ewan Milne wrote:
> > On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
> >> The current ALUA device_handler has two drawbacks:
> >> - We're sending a 'SET TARGET PORT GROUP' command to every LUN,
> >>   disregarding the fact that several LUNs might be in a port group
> >>   and will be automatically switched whenever _any_ LUN within
> >>   that port group receives the command.
> >> - Whenever a LUN is in 'transitioning' mode we cannot block I/O
> >>   to that LUN, instead the controller has to abort the command.
> >>   This leads to increased traffic across the wire and heavy load
> >>   on the controller during switchover.
> > 
> > I'm not sure I understand what this means - why couldn't we block I/O?
> > and what does 'heavy load' mean?  Aborting commands is 'heavy load'?
> > 
> If we're getting a sense code indicating that the LUN is in
> transitioning _and_ we're blocking I/O we never ever send down I/Os to
> that driver anymore, so we cannot receive any sense codes indicating the
> transitioning is done.
> At the same time, every I/O we're sending down will be returned by the
> storage I/O with a sense code, requiring us to retry the command.
> Hence we're constantly retrying I/O.

Ah, OK.  Perhaps including this explanation either in the comments with
patch 22/23 which adds the TEST UNIT READY commands to poll for the
status, or in the patch description somewhere would be helpful.

> 
> [ .. ]
> >> @@ -811,10 +1088,17 @@ failed:
> >>  static void alua_bus_detach(struct scsi_device *sdev)
> >>  {
> >>  	struct alua_dh_data *h = sdev->handler_data;
> >> +	struct alua_port_group *pg;
> >>  
> >> -	if (h->pg) {
> >> -		kref_put(&h->pg->kref, release_port_group);
> >> -		h->pg = NULL;
> >> +	spin_lock(&h->pg_lock);
> >> +	pg = h->pg;
> >> +	rcu_assign_pointer(h->pg, NULL);
> >> +	spin_unlock(&h->pg_lock);
> >> +	synchronize_rcu();
> >> +	if (pg) {
> >> +		if (pg->rtpg_sdev)
> >> +			flush_workqueue(pg->work_q);
> >> +		kref_put(&pg->kref, release_port_group);
> >>  	}
> >>  	sdev->handler_data = NULL;
> >>  	kfree(h);
> > 
> > So, you've already had a bit of discussion with Christoph about this,
> > the main portion of your ALUA rewrite, and I won't go over all of that,
> > except to say that I'd have to agree that having separate work queues
> > for the different RTPG/STPG functions and having them manipulate each
> > other's flags seems like we'd be better off having just one work
> > function that did everything.  Less messy and easier to maintain.
> > 
> > Also, it seems like wrong ordering of kref_get() vs. scsi_device_get(),
> > in alua_rtpg_queue() since they are released as kref_put() then
> > scsi_device_put()?
> > 
> Yeah, I've reworked the reference counting.
> And reverted the workqueue handling to use the original model.
> 
> Cheers,
> 
> Hannes

Re: [PATCHv4 00/23] asynchronous ALUA device handler

[Bart Van Assche]

On 08/27/2015 05:41 AM, Hannes Reinecke wrote:
> here is an update to the ALUA device handler. 

Hello Hannes,

Has this patch series been tested on an initiator system with
kernel debugging enabled ? The message below appears if I boot
a kernel on which this patch series has been applied. Sorry that
I had not found the time before to test this patch series.

=================================
[ INFO: inconsistent lock state ]
sd 23:0:0:66: Attached scsi generic sg55 type 0
4.2.0-rc8-debug+ #1 Not tainted
---------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
ksoftirqd/1/13 [HC1[1]:SC1[1]:HE0:SE0] takes:
 (&(&pg->rtpg_lock)->rlock){?.+...}, at: [<ffffffffa040b78c>] alua_rtpg_queue+0x4c/0x160 [scsi_dh_alua]
{HARDIRQ-ON-W} state was registered at:
  [<ffffffff810be862>] __lock_acquire+0xae2/0x1c70
  [<ffffffff810c027b>] lock_acquire+0xcb/0x290
  [<ffffffff815070e8>] _raw_spin_lock+0x38/0x50
  [<ffffffffa040c5da>] alua_check_vpd+0x7da/0x860 [scsi_dh_alua]
  [<ffffffffa040cfee>] alua_initialize+0xfe/0x340 [scsi_dh_alua]
  [<ffffffffa040d2de>] alua_bus_attach+0x8e/0xc0 [scsi_dh_alua]
  [<ffffffffa0026261>] scsi_dh_handler_attach+0x31/0x90 [scsi_mod]
  [<ffffffffa0026683>] scsi_dh_attach+0x93/0xa0 [scsi_mod]
  [<ffffffffa01298d2>] multipath_ctr+0x8e2/0xaec [dm_multipath]
  [<ffffffffa041a324>] dm_table_add_target+0x124/0x370 [dm_mod]
  [<ffffffffa041d681>] table_load+0x121/0x350 [dm_mod]
  [<ffffffffa041e259>] ctl_ioctl+0x299/0x520 [dm_mod]
  [<ffffffffa041e4f3>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
  [<ffffffff811d1d8d>] do_vfs_ioctl+0x30d/0x580
  [<ffffffff811d2041>] SyS_ioctl+0x41/0x70
  [<ffffffff81507cf2>] entry_SYSCALL_64_fastpath+0x16/0x7a
irq event stamp: 940607
hardirqs last  enabled at (940606): [<ffffffff81507366>] _raw_spin_unlock_irqrestore+0x36/0x60
hardirqs last disabled at (940607): [<ffffffff815088ab>] common_interrupt+0x6b/0x70
softirqs last  enabled at (940536): [<ffffffff8107a035>] __do_softirq+0x345/0x5d0
softirqs last disabled at (940541): [<ffffffff8107a2e5>] run_ksoftirqd+0x25/0x70

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&(&pg->rtpg_lock)->rlock);
  <Interrupt>
    lock(&(&pg->rtpg_lock)->rlock);

 *** DEADLOCK ***

1 lock held by ksoftirqd/1/13:
 #0:  (rcu_callback){......}, at: [<ffffffff810d6716>] rcu_process_callbacks+0x2b6/0xa50

stack backtrace:
CPU: 1 PID: 13 Comm: ksoftirqd/1 Not tainted 4.2.0-rc8-debug+ #1
Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
 ffffffff824a0c90 ffff88047fc43918 ffffffff814ff6c4 0000000000000000
 ffff88045c66ac80 ffff88047fc43978 ffffffff814fc5a0 0000000000000001
 ffff880400000000 ffff880400000000 ffffffff8102bf6f 0000000000000000
Call Trace:
 <IRQ>  [<ffffffff814ff6c4>] dump_stack+0x4c/0x65
 [<ffffffff814fc5a0>] print_usage_bug+0x1f2/0x203
 [<ffffffff8102bf6f>] ? save_stack_trace+0x2f/0x50
 [<ffffffff810bcda0>] ? check_usage_backwards+0x110/0x110
 [<ffffffff810bd762>] mark_lock+0x212/0x2a0
 [<ffffffff810be997>] __lock_acquire+0xc17/0x1c70
 [<ffffffff810be29d>] ? __lock_acquire+0x51d/0x1c70
 [<ffffffff810c027b>] lock_acquire+0xcb/0x290
 [<ffffffffa040b78c>] ? alua_rtpg_queue+0x4c/0x160 [scsi_dh_alua]
 [<ffffffff81507a50>] _raw_spin_lock_irqsave+0x50/0x70
 [<ffffffffa040b78c>] ? alua_rtpg_queue+0x4c/0x160 [scsi_dh_alua]
 [<ffffffffa040b78c>] alua_rtpg_queue+0x4c/0x160 [scsi_dh_alua]
 [<ffffffffa040bbed>] alua_check+0xdd/0x270 [scsi_dh_alua]
 [<ffffffffa040bb15>] ? alua_check+0x5/0x270 [scsi_dh_alua]
 [<ffffffffa040bde6>] alua_check_sense+0x66/0x80 [scsi_dh_alua]
 [<ffffffffa0016546>] scsi_check_sense+0x76/0x3e0 [scsi_mod]
 [<ffffffffa0018c4f>] scsi_decide_disposition+0x17f/0x1f0 [scsi_mod]
 [<ffffffffa001cfe7>] scsi_softirq_done+0x57/0x150 [scsi_mod]
 [<ffffffff8127a3ba>] __blk_mq_complete_request+0x8a/0x110
 [<ffffffff8127a456>] blk_mq_complete_request+0x16/0x20
 [<ffffffffa001c048>] scsi_mq_done+0x48/0x1b0 [scsi_mod]
 [<ffffffffa03d10d9>] srp_recv_completion+0x249/0x720 [ib_srp]
 [<ffffffffa02c13b7>] mlx4_ib_cq_comp+0x17/0x20 [mlx4_ib]
 [<ffffffffa00bd8e8>] mlx4_cq_completion+0x38/0x70 [mlx4_core]
 [<ffffffffa00bf1b3>] mlx4_eq_int+0x493/0xcb0 [mlx4_core]
 [<ffffffff810bbbdd>] ? __lock_is_held+0x4d/0x70
 [<ffffffffa00bfa74>] mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core]
 [<ffffffff810cb030>] handle_irq_event_percpu+0x40/0x4b0
 [<ffffffff810cb4e4>] handle_irq_event+0x44/0x70
 [<ffffffff810ce470>] handle_edge_irq+0x90/0x140
 [<ffffffff8101e3d2>] handle_irq+0x22/0x40
 [<ffffffff8150aa11>] do_IRQ+0x51/0xe0
 [<ffffffff811d493c>] ? __d_free+0x1c/0x20
 [<ffffffff815088b0>] common_interrupt+0x70/0x70
 <EOI>  [<ffffffff811ad1d1>] ? set_track+0x61/0x100
 [<ffffffff8150736b>] ? _raw_spin_unlock_irqrestore+0x3b/0x60
 [<ffffffff814fe65d>] __slab_free+0x56/0x1ff
 [<ffffffff811d493c>] ? __d_free+0x1c/0x20
 [<ffffffff811b9987>] ? put_object+0x37/0x60
 [<ffffffff811af9d2>] ? kmem_cache_free+0xb2/0x370
 [<ffffffff811d493c>] ? __d_free+0x1c/0x20
 [<ffffffff811d493c>] ? __d_free+0x1c/0x20
 [<ffffffff811afc7f>] kmem_cache_free+0x35f/0x370
 [<ffffffff811d4920>] ? __d_free_external+0x40/0x40
 [<ffffffff811d493c>] __d_free+0x1c/0x20
 [<ffffffff810d6756>] rcu_process_callbacks+0x2f6/0xa50
 [<ffffffff810d6716>] ? rcu_process_callbacks+0x2b6/0xa50
 [<ffffffff81079dc2>] __do_softirq+0xd2/0x5d0
 [<ffffffff8107a2e5>] run_ksoftirqd+0x25/0x70
 [<ffffffff8109b279>] smpboot_thread_fn+0x139/0x200
 [<ffffffff8109b140>] ? sort_range+0x30/0x30
 [<ffffffff81097878>] kthread+0xf8/0x110
 [<ffffffff81097780>] ? kthread_create_on_node+0x210/0x210
 [<ffffffff8150815f>] ret_from_fork+0x3f/0x70
 [<ffffffff81097780>] ? kthread_create_on_node+0x210/0x210

Re: [PATCH 10/23] scsi_dh_alua: Pass buffer as function argument

[Hannes Reinecke]

On 09/22/2015 08:43 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> Pass in the buffer as a function argument for submit_rtpg().
>>
>> Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 20 ++++++++++----------
>>  1 file changed, 10 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index 0636721..9e2b3af 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -137,12 +137,13 @@ static struct request *get_alua_req(struct scsi_device *sdev,
>>   * submit_rtpg - Issue a REPORT TARGET GROUP STATES command
>>   * @sdev: sdev the command should be sent to
>>   */
>> -static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
>> +static unsigned submit_rtpg(struct scsi_device *sdev, unsigned char *buff,
>> +			    int bufflen, unsigned char *sense, int flags)
>>  {
>>  	struct request *rq;
>>  	int err = 0;
>>  
>> -	rq = get_alua_req(sdev, h->buff, h->bufflen, READ);
>> +	rq = get_alua_req(sdev, buff, bufflen, READ);
>>  	if (!rq) {
>>  		err = DRIVER_BUSY << 24;
>>  		goto done;
>> @@ -150,22 +151,21 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
>>  
>>  	/* Prepare the command. */
>>  	rq->cmd[0] = MAINTENANCE_IN;
>> -	if (!(h->flags & ALUA_RTPG_EXT_HDR_UNSUPP))
>> +	if (!(flags & ALUA_RTPG_EXT_HDR_UNSUPP))
>>  		rq->cmd[1] = MI_REPORT_TARGET_PGS | MI_EXT_HDR_PARAM_FMT;
>>  	else
>>  		rq->cmd[1] = MI_REPORT_TARGET_PGS;
>> -	put_unaligned_be32(h->bufflen, &rq->cmd[6]);
>> +	put_unaligned_be32(bufflen, &rq->cmd[6]);
>>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_IN);
>>  
>> -	rq->sense = h->sense;
>> +	rq->sense = sense;
>>  	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
>> -	rq->sense_len = h->senselen = 0;
>> +	rq->sense_len = 0;
>>  
>>  	blk_execute_rq(rq->q, NULL, rq, 1);
>> -	if (rq->errors) {
>> +	if (rq->errors)
>>  		err = rq->errors;
>> -		h->senselen = rq->sense_len;
>> -	}
>> +
>>  	blk_put_request(rq);
>>  done:
>>  	return err;
>> @@ -491,7 +491,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>>  		expiry = round_jiffies_up(jiffies + h->transition_tmo * HZ);
>>  
>>   retry:
>> -	retval = submit_rtpg(sdev, h);
>> +	retval = submit_rtpg(sdev, h->buff, h->bufflen, h->sense, h->flags);
>>  
>>  	if (retval) {
>>  		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
> 
> This patch also removes the setting of h->senselen, you should mention
> that in the patch description.  Or, perhaps the removal of h->senselen
> should have been done as part of patch 12/23, which removed the senselen
> field (and the sense buffer field) from the alua_dh_data structure.

Is already done with the next version of the patchset.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/23] scsi_dh_alua: Make stpg synchronous

[Hannes Reinecke]

On 09/22/2015 08:50 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> We should be issuing STPG synchronously as we need to
>> evaluate the return code on failure.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 179 +++++++++++++----------------
>>  1 file changed, 83 insertions(+), 96 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index 9e2b3af..fd0385e 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -172,76 +172,28 @@ done:
>>  }
>>  
>>  /*
>> - * stpg_endio - Evaluate SET TARGET GROUP STATES
>> - * @sdev: the device to be evaluated
>> - * @state: the new target group state
>> - *
>> - * Evaluate a SET TARGET GROUP STATES command response.
>> - */
>> -static void stpg_endio(struct request *req, int error)
>> -{
>> -	struct alua_dh_data *h = req->end_io_data;
>> -	struct scsi_sense_hdr sense_hdr;
>> -	unsigned err = SCSI_DH_OK;
>> -
>> -	if (host_byte(req->errors) != DID_OK ||
>> -	    msg_byte(req->errors) != COMMAND_COMPLETE) {
>> -		err = SCSI_DH_IO;
>> -		goto done;
>> -	}
>> -
>> -	if (scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
>> -				 &sense_hdr)) {
>> -		err = alua_check_sense(h->sdev, &sense_hdr);
>> -		if (err == ADD_TO_MLQUEUE) {
>> -			err = SCSI_DH_RETRY;
>> -			goto done;
>> -		}
>> -		sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
>> -			    ALUA_DH_NAME);
>> -		scsi_print_sense_hdr(h->sdev, ALUA_DH_NAME, &sense_hdr);
>> -		err = SCSI_DH_IO;
>> -	} else if (error)
>> -		err = SCSI_DH_IO;
>> -
>> -	if (err == SCSI_DH_OK) {
>> -		h->state = TPGS_STATE_OPTIMIZED;
>> -		sdev_printk(KERN_INFO, h->sdev,
>> -			    "%s: port group %02x switched to state %c\n",
>> -			    ALUA_DH_NAME, h->group_id,
>> -			    print_alua_state(h->state));
>> -	}
>> -done:
>> -	req->end_io_data = NULL;
>> -	__blk_put_request(req->q, req);
>> -	if (h->callback_fn) {
>> -		h->callback_fn(h->callback_data, err);
>> -		h->callback_fn = h->callback_data = NULL;
>> -	}
>> -	return;
>> -}
>> -
>> -/*
>>   * submit_stpg - Issue a SET TARGET GROUP STATES command
>>   *
>>   * Currently we're only setting the current target port group state
>>   * to 'active/optimized' and let the array firmware figure out
>>   * the states of the remaining groups.
>>   */
>> -static unsigned submit_stpg(struct alua_dh_data *h)
>> +static unsigned submit_stpg(struct scsi_device *sdev, int group_id,
>> +			    unsigned char *sense)
>>  {
>>  	struct request *rq;
>> +	unsigned char stpg_data[8];
>>  	int stpg_len = 8;
>> -	struct scsi_device *sdev = h->sdev;
>> +	int err = 0;
>>  
>>  	/* Prepare the data buffer */
>> -	memset(h->buff, 0, stpg_len);
>> -	h->buff[4] = TPGS_STATE_OPTIMIZED & 0x0f;
>> -	put_unaligned_be16(h->group_id, &h->buff[6]);
>> +	memset(stpg_data, 0, stpg_len);
>> +	stpg_data[4] = TPGS_STATE_OPTIMIZED & 0x0f;
>> +	put_unaligned_be16(group_id, &stpg_data[6]);
>>  
>> -	rq = get_alua_req(sdev, h->buff, stpg_len, WRITE);
>> +	rq = get_alua_req(sdev, stpg_data, stpg_len, WRITE);
>>  	if (!rq)
>> -		return SCSI_DH_RES_TEMP_UNAVAIL;
>> +		return DRIVER_BUSY << 24;
>>  
>>  	/* Prepare the command. */
>>  	rq->cmd[0] = MAINTENANCE_OUT;
>> @@ -249,13 +201,17 @@ static unsigned submit_stpg(struct alua_dh_data *h)
>>  	put_unaligned_be32(stpg_len, &rq->cmd[6]);
>>  	rq->cmd_len = COMMAND_SIZE(MAINTENANCE_OUT);
>>  
>> -	rq->sense = h->sense;
>> +	rq->sense = sense;
>>  	memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
>> -	rq->sense_len = h->senselen = 0;
>> -	rq->end_io_data = h;
>> +	rq->sense_len = 0;
>> +
>> +	blk_execute_rq(rq->q, NULL, rq, 1);
>> +	if (rq->errors)
>> +		err = rq->errors;
>> +
>> +	blk_put_request(rq);
>>  
>> -	blk_execute_rq_nowait(rq->q, NULL, rq, 1, stpg_endio);
>> -	return SCSI_DH_OK;
>> +	return err;
>>  }
>>  
>>  /*
>> @@ -619,6 +575,68 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_
>>  }
>>  
>>  /*
>> + * alua_stpg - Issue a SET TARGET GROUP STATES command
>> + *
>> + * Issue a SET TARGET GROUP STATES command and evaluate the
>> + * response. Returns SCSI_DH_RETRY per default to trigger
>> + * a re-evaluation of the target group state.
>> + */
>> +static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)
>> +{
>> +	int retval;
>> +	struct scsi_sense_hdr sense_hdr;
>> +
>> +	if (!(h->tpgs & TPGS_MODE_EXPLICIT)) {
>> +		/* Only implicit ALUA supported, retry */
>> +		return SCSI_DH_RETRY;
>> +	}
>> +	switch (h->state) {
>> +	case TPGS_STATE_OPTIMIZED:
>> +		return SCSI_DH_OK;
>> +	case TPGS_STATE_NONOPTIMIZED:
>> +		if ((h->flags & ALUA_OPTIMIZE_STPG) &&
>> +		    !h->pref &&
>> +		    (h->tpgs & TPGS_MODE_IMPLICIT))
>> +			return SCSI_DH_OK;
>> +		break;
>> +	case TPGS_STATE_STANDBY:
>> +	case TPGS_STATE_UNAVAILABLE:
>> +		break;
>> +	case TPGS_STATE_OFFLINE:
>> +		return SCSI_DH_IO;
>> +		break;
>> +	case TPGS_STATE_TRANSITIONING:
>> +		break;
>> +	default:
>> +		sdev_printk(KERN_INFO, sdev,
>> +			    "%s: stpg failed, unhandled TPGS state %d",
>> +			    ALUA_DH_NAME, h->state);
>> +		return SCSI_DH_NOSYS;
>> +		break;
>> +	}
>> +	/* Set state to transitioning */
>> +	h->state = TPGS_STATE_TRANSITIONING;
>> +	retval = submit_stpg(sdev, h->group_id, h->sense);
>> +
>> +	if (retval) {
>> +		if (!scsi_normalize_sense(h->sense, SCSI_SENSE_BUFFERSIZE,
>> +					  &sense_hdr)) {
>> +			sdev_printk(KERN_INFO, sdev,
>> +				    "%s: stpg failed, result %d",
>> +				    ALUA_DH_NAME, retval);
>> +			if (driver_byte(retval) == DRIVER_BUSY)
>> +				return SCSI_DH_DEV_TEMP_BUSY;
>> +		} else {
>> +			sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
>> +				    ALUA_DH_NAME);
>> +			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>> +		}
>> +	}
>> +	/* Retry RTPG */
>> +	return SCSI_DH_RETRY;
>> +}
>> +
>> +/*
>>   * alua_initialize - Initialize ALUA state
>>   * @sdev: the device to be initialized
>>   *
>> @@ -695,7 +713,6 @@ static int alua_activate(struct scsi_device *sdev,
>>  {
>>  	struct alua_dh_data *h = sdev->handler_data;
>>  	int err = SCSI_DH_OK;
>> -	int stpg = 0;
>>  
>>  	err = alua_rtpg(sdev, h, 1);
>>  	if (err != SCSI_DH_OK)
>> @@ -704,39 +721,9 @@ static int alua_activate(struct scsi_device *sdev,
>>  	if (optimize_stpg)
>>  		h->flags |= ALUA_OPTIMIZE_STPG;
>>  
>> -	if (h->tpgs & TPGS_MODE_EXPLICIT) {
>> -		switch (h->state) {
>> -		case TPGS_STATE_NONOPTIMIZED:
>> -			stpg = 1;
>> -			if ((h->flags & ALUA_OPTIMIZE_STPG) &&
>> -			    (!h->pref) &&
>> -			    (h->tpgs & TPGS_MODE_IMPLICIT))
>> -				stpg = 0;
>> -			break;
>> -		case TPGS_STATE_STANDBY:
>> -		case TPGS_STATE_UNAVAILABLE:
>> -			stpg = 1;
>> -			break;
>> -		case TPGS_STATE_OFFLINE:
>> -			err = SCSI_DH_IO;
>> -			break;
>> -		case TPGS_STATE_TRANSITIONING:
>> -			err = SCSI_DH_RETRY;
>> -			break;
>> -		default:
>> -			break;
>> -		}
>> -	}
>> -
>> -	if (stpg) {
>> -		h->callback_fn = fn;
>> -		h->callback_data = data;
>> -		err = submit_stpg(h);
>> -		if (err == SCSI_DH_OK)
>> -			return 0;
>> -		h->callback_fn = h->callback_data = NULL;
>> -	}
>> -
>> +	err = alua_stpg(sdev, h);
>> +	if (err == SCSI_DH_RETRY)
>> +		err = alua_rtpg(sdev, h, 1);
>>  out:
>>  	if (fn)
>>  		fn(data, err);
> 
> submit_stpg() now returns DRIVER_BUSY << 24 instead of SCSI_DH_RES_TEMP_UNAVAIL,
> so you are changing the return code semantics here as in patch 5/23.  That's
> fine, but probably should be mentioned in the patch description.
> 
> The removed stpg_endio() code *did* evaluate the SCSI error code, and the code
> that has replaced it doesn't exactly do the same thing, so what difference in
> behavior are we going to get here?
> 
Correct. Is fixed up with the next version of the patchset.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 14/23] scsi_dh_alua: allocate RTPG buffer separately

[Hannes Reinecke]

On 09/22/2015 09:04 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> The RTPG buffer will only evaluated within alua_rtpg(),
>> so we can allocate it locally there and avoid having to
>> put it into the global structure.
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 56 +++++++++++-------------------
>>  1 file changed, 21 insertions(+), 35 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index d1010dd..4157fe2 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -56,7 +56,7 @@
>>  #define TPGS_MODE_IMPLICIT		0x1
>>  #define TPGS_MODE_EXPLICIT		0x2
>>  
>> -#define ALUA_INQUIRY_SIZE		36
>> +#define ALUA_RTPG_SIZE			128
>>  #define ALUA_FAILOVER_TIMEOUT		60
>>  #define ALUA_FAILOVER_RETRIES		5
>>  
>> @@ -75,9 +75,6 @@ struct alua_port_group {
>>  	int			state;
>>  	int			pref;
>>  	unsigned		flags; /* used for optimizing STPG */
>> -	unsigned char		inq[ALUA_INQUIRY_SIZE];
>> -	unsigned char		*buff;
>> -	int			bufflen;
>>  	unsigned char		transition_tmo;
>>  };
>>  
>> @@ -96,21 +93,6 @@ struct alua_dh_data {
>>  static char print_alua_state(int);
>>  static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
>>  
>> -static int realloc_buffer(struct alua_port_group *pg, unsigned len)
>> -{
>> -	if (pg->buff && pg->buff != pg->inq)
>> -		kfree(pg->buff);
>> -
>> -	pg->buff = kmalloc(len, GFP_NOIO);
>> -	if (!pg->buff) {
>> -		pg->buff = pg->inq;
>> -		pg->bufflen = ALUA_INQUIRY_SIZE;
>> -		return 1;
>> -	}
>> -	pg->bufflen = len;
>> -	return 0;
>> -}
>> -
>>  static void release_port_group(struct kref *kref)
>>  {
>>  	struct alua_port_group *pg;
>> @@ -120,8 +102,6 @@ static void release_port_group(struct kref *kref)
>>  	spin_lock(&port_group_lock);
>>  	list_del(&pg->node);
>>  	spin_unlock(&port_group_lock);
>> -	if (pg->buff && pg->inq != pg->buff)
>> -		kfree(pg->buff);
>>  	kfree(pg);
>>  }
>>  
>> @@ -300,8 +280,6 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>>  		return SCSI_DH_DEV_TEMP_BUSY;
>>  	}
>>  	pg->group_id = group_id;
>> -	pg->buff = pg->inq;
>> -	pg->bufflen = ALUA_INQUIRY_SIZE;
>>  	pg->tpgs = h->tpgs;
>>  	pg->state = TPGS_STATE_OPTIMIZED;
>>  	kref_init(&pg->kref);
>> @@ -424,8 +402,8 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wait_for_transition)
>>  {
>>  	struct scsi_sense_hdr sense_hdr;
>> -	int len, k, off, valid_states = 0;
>> -	unsigned char *ucp;
>> +	int len, k, off, valid_states = 0, bufflen = ALUA_RTPG_SIZE;
>> +	unsigned char *ucp, *buff;
>>  	unsigned err, retval;
>>  	unsigned long expiry, interval = 0;
>>  	unsigned int tpg_desc_tbl_off;
>> @@ -436,9 +414,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  	else
>>  		expiry = round_jiffies_up(jiffies + pg->transition_tmo * HZ);
>>  
>> +	buff = kzalloc(bufflen, GFP_KERNEL);
>> +	if (!buff)
>> +		return SCSI_DH_DEV_TEMP_BUSY;
>> +
>>   retry:
>> -	retval = submit_rtpg(sdev, pg->buff, pg->bufflen,
>> -			     &sense_hdr, pg->flags);
>> +	retval = submit_rtpg(sdev, buff, bufflen, &sense_hdr, pg->flags);
>>  
>>  	if (retval) {
>>  		if (!scsi_sense_valid(&sense_hdr)) {
>> @@ -449,6 +430,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  				err = SCSI_DH_DEV_TEMP_BUSY;
>>  			else
>>  				err = SCSI_DH_IO;
>> +			kfree(buff);
>>  			return err;
>>  		}
>>  
>> @@ -477,14 +459,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  		sdev_printk(KERN_ERR, sdev, "%s: rtpg failed\n",
>>  			    ALUA_DH_NAME);
>>  		scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
>> +		kfree(buff);
>>  		return SCSI_DH_IO;
>>  	}
>>  
>> -	len = get_unaligned_be32(&pg->buff[0]) + 4;
>> +	len = get_unaligned_be32(&buff[0]) + 4;
>>  
>> -	if (len > pg->bufflen) {
>> +	if (len > bufflen) {
>>  		/* Resubmit with the correct length */
>> -		if (realloc_buffer(pg, len)) {
>> +		kfree(buff);
>> +		bufflen = len;
>> +		buff = kmalloc(bufflen, GFP_KERNEL);
>> +		if (!buff) {
>>  			sdev_printk(KERN_WARNING, sdev,
>>  				    "%s: kmalloc buffer failed\n",__func__);
>>  			/* Temporary failure, bypass */
>> @@ -494,9 +480,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  	}
>>  
>>  	orig_transition_tmo = pg->transition_tmo;
>> -	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR &&
>> -	    pg->buff[5] != 0)
>> -		pg->transition_tmo = pg->buff[5];
>> +	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR && buff[5] != 0)
>> +		pg->transition_tmo = buff[5];
>>  	else
>>  		pg->transition_tmo = ALUA_FAILOVER_TIMEOUT;
>>  
>> @@ -508,12 +493,12 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  		expiry = jiffies + pg->transition_tmo * HZ;
>>  	}
>>  
>> -	if ((pg->buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
>> +	if ((buff[4] & RTPG_FMT_MASK) == RTPG_FMT_EXT_HDR)
>>  		tpg_desc_tbl_off = 8;
>>  	else
>>  		tpg_desc_tbl_off = 4;
>>  
>> -	for (k = tpg_desc_tbl_off, ucp = pg->buff + tpg_desc_tbl_off;
>> +	for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off;
>>  	     k < len;
>>  	     k += off, ucp += off) {
>>  
>> @@ -563,6 +548,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  		err = SCSI_DH_OK;
>>  		break;
>>  	}
>> +	kfree(buff);
>>  	return err;
>>  }
>>  
> 
> I guess.  Are these buffers so big that we don't want to make it part of
> a structure that just stays allocated?
> 
The point is that the buffer scales with the size of the installation,
so we cannot allocate it statically.

> We should have some validation on the length of the data returned by RTPG in
> the case where the buffer is re-kmalloc'ed, we can't kmalloc 4GB - 1 + 4 bytes.
> And if the length is too big, it is not a "temporary" failure.
Well, it's hard to distinguish between ENOMEM because of a (temporary)
memory shortage and ENOMEM because kmalloc cannot allocate that much
memory in general. Mind you, FC-AL can only have up to 128M bytes worth
of data here (64k possible loop IDs with 8 bytes port IDs each).

However, during every testing I've done (and this particular code is
running in SLES for some years now) I've never ever seen an issue with
kmalloc failing to allocate memory.

> We should maybe only retry once with a larger buffer, rather than potentially
> loop forever if the device keeps returning different information.
> 
> (I suppose we have the same potential issue with REPORT LUNS as well.)
> 
Well, the size shouldn't really change (unless someone does an array
reconfiguration). So at most we'll be retrying once (plus any sense code
retries, of course).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 15/23] scsi_dh_alua: simplify sense code handling

[Hannes Reinecke]

On 09/22/2015 09:10 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> Most sense code is already handled in the generic
>> code, so we shouldn't be adding special cases here.
>> However, when doing so we need to check for
>> unit attention whenever we're sending an internal
>> command.
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 50 +++++++-----------------------
>>  1 file changed, 11 insertions(+), 39 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index 4157fe2..dbe9ff2 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -91,7 +91,6 @@ struct alua_dh_data {
>>  #define ALUA_POLICY_SWITCH_ALL		1
>>  
>>  static char print_alua_state(int);
>> -static int alua_check_sense(struct scsi_device *, struct scsi_sense_hdr *);
>>  
>>  static void release_port_group(struct kref *kref)
>>  {
>> @@ -323,28 +322,6 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  			 * LUN Not Accessible - ALUA state transition
>>  			 */
>>  			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0b)
>> -			/*
>> -			 * LUN Not Accessible -- Target port in standby state
>> -			 */
>> -			return SUCCESS;
>> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x0c)
>> -			/*
>> -			 * LUN Not Accessible -- Target port in unavailable state
>> -			 */
>> -			return SUCCESS;
>> -		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x12)
>> -			/*
>> -			 * LUN Not Ready -- Offline
>> -			 */
>> -			return SUCCESS;
>> -		if (sdev->allow_restart &&
>> -		    sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x02)
>> -			/*
>> -			 * if the device is not started, we need to wake
>> -			 * the error handler to start the motor
>> -			 */
>> -			return FAILED;
>>  		break;
>>  	case UNIT_ATTENTION:
>>  		if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
>> @@ -359,7 +336,7 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  			return ADD_TO_MLQUEUE;
>>  		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x01)
>>  			/*
>> -			 * Mode Parameters Changed
>> +			 * Mode Parameter Changed
> 
> See below.
> 
>>  			 */
>>  			return ADD_TO_MLQUEUE;
>>  		if (sense_hdr->asc == 0x2a && sense_hdr->ascq == 0x06)
>> @@ -372,18 +349,6 @@ static int alua_check_sense(struct scsi_device *sdev,
>>  			 * Implicit ALUA state transition failed
>>  			 */
>>  			return ADD_TO_MLQUEUE;
>> -		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x03)
>> -			/*
>> -			 * Inquiry data has changed
>> -			 */
>> -			return ADD_TO_MLQUEUE;
> 
> ??? See below.
> 
>> -		if (sense_hdr->asc == 0x3f && sense_hdr->ascq == 0x0e)
>> -			/*
>> -			 * REPORTED_LUNS_DATA_HAS_CHANGED is reported
>> -			 * when switching controllers on targets like
>> -			 * Intel Multi-Flex. We can just retry.
>> -			 */
>> -			return ADD_TO_MLQUEUE;
> 
> ??? See below.
> 
>>  		break;
>>  	}
>>  
>> @@ -448,9 +413,16 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int w
>>  			pg->flags |= ALUA_RTPG_EXT_HDR_UNSUPP;
>>  			goto retry;
>>  		}
>> -
>> -		err = alua_check_sense(sdev, &sense_hdr);
>> -		if (err == ADD_TO_MLQUEUE && time_before(jiffies, expiry)) {
>> +		/*
>> +		 * Retry on ALUA state transition or if any
>> +		 * UNIT ATTENTION occurred.
>> +		 */
>> +		if (sense_hdr.sense_key == NOT_READY &&
>> +		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
>> +			err = SCSI_DH_RETRY;
>> +		else if (sense_hdr.sense_key == UNIT_ATTENTION)
>> +			err = SCSI_DH_RETRY;
>> +		if (err == SCSI_DH_RETRY && time_before(jiffies, expiry)) {
>>  			sdev_printk(KERN_ERR, sdev, "%s: rtpg retry\n",
>>  				    ALUA_DH_NAME);
>>  			scsi_print_sense_hdr(sdev, ALUA_DH_NAME, &sense_hdr);
> 
> "Mode Parameters Changed" comment should not have been changed to "Mode Parameter Changed".
> The actual T10 text is "Mode Parameters Changed", so leave it the way it is.
> 
Okay.

> The sense code handling in the UA case for ASC/ASCQ 3F 03 is changed by this patch to
> return SUCCESS from scsi_check_sense() instead of ADD_TO_MLQUEUE from alua_check_sense(),
> and the sense code handling in the UA case for ASC/ASCQ 3F 0E is changed by this patch to
> return NEEDS_RETRY from scsi_check_sense() instead of ADD_TO_MLQUEUE from alua_check_sense().
> So we will not reissue the command on INQUIRY DATA HAS CHANGED and will have different
> flow of control on REPORTED LUNS DATA HAS CHANGED.  What is the reason for this?
> 
'INQUIRY DATA HAS CHANGED' will be handled with a later patch, so I
had it removed from here. But indeed, these hunks shouldn't be
removed. I'll be reverting that.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 16/23] scsi: Add scsi_vpd_lun_id()

[Hannes Reinecke]

On 09/22/2015 09:17 PM, Ewan Milne wrote:
[ .. ]
> 
> scsi_vpd_lun_id() is an exported function, but does not check if sdev->vpd_pg83 != NULL
> (i.e. no VPD 83 info), and we will crash if it is NULL.  Also, why return -EAGAIN?
> What will be different next time?  Maybe a different errno would be better, like
> -ENOENT or -ENODATA, I'm not sure.
> 
> In the SCSI name string case, if the name has to be truncated to fit the supplied buffer,
> scsi_vpd_lun_id() does not ensure that the last byte is zero, it relies on the caller to
> have done this, (which alua_check_vpd() does not do).  I'm not sure if we should truncate
> the SCSI name string here or return an error here, as a truncated name will likely not
> be unique.
> 
I've now updated the description, and decreased the priority if the
SCSI name descriptor is truncated. With that any other descriptor
will have preference if the SCSI name descriptor is truncated.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 17/23] scsi_dh_alua: use unique device id

[Hannes Reinecke]

On 09/22/2015 09:31 PM, Ewan Milne wrote:
> On Thu, 2015-08-27 at 14:41 +0200, Hannes Reinecke wrote:
>> Use scsi_vpd_lun_id() to assign a unique device identification
>> to the alua port group structure.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>  drivers/scsi/device_handler/scsi_dh_alua.c | 70 +++++++++++++++++++++++++++---
>>  1 file changed, 65 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/scsi/device_handler/scsi_dh_alua.c b/drivers/scsi/device_handler/scsi_dh_alua.c
>> index dbe9ff2..c2b2100b 100644
>> --- a/drivers/scsi/device_handler/scsi_dh_alua.c
>> +++ b/drivers/scsi/device_handler/scsi_dh_alua.c
>> @@ -70,6 +70,8 @@ static DEFINE_SPINLOCK(port_group_lock);
>>  struct alua_port_group {
>>  	struct kref		kref;
>>  	struct list_head	node;
>> +	unsigned char		device_id_str[256];
>> +	int			device_id_size;
> 
> I prefer _len instead of _size, _size should refer to the size of the buffer,
> not the current length of the data in it.
> 
>>  	int			group_id;
>>  	int			tpgs;
>>  	int			state;
>> @@ -229,7 +231,9 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>>  {
>>  	unsigned char *d;
>>  	int group_id = -1;
>> -	struct alua_port_group *pg = NULL;
>> +	char device_id_str[256];
>> +	int device_id_size;
>> +	struct alua_port_group *tmp_pg, *pg = NULL;
>>  
>>  	if (!sdev->vpd_pg83)
>>  		return SCSI_DH_DEV_UNSUPP;
>> @@ -266,9 +270,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>>  		h->tpgs = TPGS_MODE_NONE;
>>  		return SCSI_DH_DEV_UNSUPP;
>>  	}
>> +	device_id_size = scsi_vpd_lun_id(sdev, device_id_str, 256);
> 
> should be sizeof(device_id_str) not hardcoded 256
> 
Okay.
>> +	if (device_id_size <= 0) {
>> +		/*
>> +		 * Internal error: TPGS supported by no
> 
> "Internal error: TPGS supported by no"  should be "but no"
> 
Fixed now.

>> +		 * device identifcation found.
>> +		 * Disable ALUA support.
>> +		 */
>> +		sdev_printk(KERN_INFO, sdev,
>> +			    "%s: No device descriptors found\n",
>> +			    ALUA_DH_NAME);
>> +		h->tpgs = TPGS_MODE_NONE;
>> +		return SCSI_DH_DEV_UNSUPP;
>> +	}
>>  	sdev_printk(KERN_INFO, sdev,
>> -		    "%s: port group %02x rel port %02x\n",
>> -		    ALUA_DH_NAME, group_id, h->rel_port);
>> +		    "%s: device %s port group %02x "
>> +		    "rel port %02x\n", ALUA_DH_NAME,
>> +		    device_id_str, group_id, h->rel_port);
>> +	spin_lock(&port_group_lock);
>> +	list_for_each_entry(tmp_pg, &port_group_list, node) {
>> +		if (tmp_pg->group_id != group_id)
>> +			continue;
>> +		if (tmp_pg->device_id_size != device_id_size)
>> +			continue;
>> +		if (strncmp(tmp_pg->device_id_str, device_id_str,
>> +			    device_id_size))
>> +			continue;
>> +		h->pg = tmp_pg;
>> +		kref_get(&tmp_pg->kref);
>> +		break;
>> +	}
>> +	spin_unlock(&port_group_lock);
>> +	if (h->pg)
>> +		return SCSI_DH_OK;
> 
> The lookup checks whether h->pg == NULL but the function never
> explicitly sets it to NULL before iterating.
> 
For my next iteration I've reworked this so that h->pg is explicitly
set.

>>  
>>  	pg = kzalloc(sizeof(struct alua_port_group), GFP_KERNEL);
>>  	if (!pg) {
>> @@ -278,13 +312,39 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h)
>>  		/* Temporary failure, bypass */
>>  		return SCSI_DH_DEV_TEMP_BUSY;
>>  	}
>> +	if (device_id_size)
>> +		strncpy(pg->device_id_str, device_id_str, 256);
> 
> should be sizeof(device_id_str) not hardcoded 256
> 
Okay.

>> +	else
>> +		pg->device_id_str[0] = '\0';
>> +
>> +	pg->device_id_size = device_id_size;
>>  	pg->group_id = group_id;
>>  	pg->tpgs = h->tpgs;
>>  	pg->state = TPGS_STATE_OPTIMIZED;
>>  	kref_init(&pg->kref);
>>  	spin_lock(&port_group_lock);
>> -	list_add(&pg->node, &port_group_list);
>> -	h->pg = pg;
>> +	/*
>> +	 * Re-check list again to catch
>> +	 * concurrent updates
>> +	 */
>> +	list_for_each_entry(tmp_pg, &port_group_list, node) {
>> +		if (tmp_pg->group_id != pg->group_id)
>> +			continue;
>> +		if (tmp_pg->device_id_size != pg->device_id_size)
>> +			continue;
>> +		if (strncmp(tmp_pg->device_id_str, pg->device_id_str,
>> +			    device_id_size))
>> +			continue;
>> +		h->pg = tmp_pg;
>> +		kref_get(&tmp_pg->kref);
>> +		kfree(pg);
> 
> With the added check for an existing alua_port_group object, and the kfree() of
> the alua_port_group that had been allocated if an existing one is found, the code does not
> do a destroy_workqueue() on pg->work_q.  
> 
With the current rework I've removed the per-pg workqueues, so that
issue doesn't occur anymore.

>> +		pg = NULL;
>> +		break;
>> +	}
>> +	if (pg) {
>> +		list_add(&pg->node, &port_group_list);
>> +		h->pg = pg;
>> +	}
>>  	spin_unlock(&port_group_lock);
>>  
>>  	return SCSI_DH_OK;
>  
> An explanation in the comments about the ALUA topology and what the device_id vs.
> the group_id represents might be helpful.  It occurred to me that if someone doesn't
> understand that you can have the same device_id behind different port groups they
> won't understand this code.
> 
> HOST  ---->     STORAGE CTRL PG 1   ---->    LUN X
>       ---->                         ---->    LUN X
>       ---->     STORAGE CTRL PG 2   ---->    LUN X
>       ---->                         ---->    LUN X
> 
Hmm. Someone willing to understand this code should be reasonable
familiar with SPC, so I doubt that'll be an issue.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 19/23] scsi_dh_alua: Use workqueue for RTPG

[Todd Gill]

On 09/02/2015 04:48 AM, Hannes Reinecke wrote:
> On 09/02/2015 08:39 AM, Christoph Hellwig wrote:
>> On Tue, Sep 01, 2015 at 02:57:57PM +0200, Hannes Reinecke wrote:
>>
>> It might be a good idea to prioritize that.  Todd has been asking
>> for multipath monitoring APIs and suggested adding D-BUS APIs to
>> multipathd.  I'd much prefer them directly talking to the kernel
>> instead of cementing multipathd, which is a bit of a wart.

I have a working prototype of a D-Bus API implemented as a new thread of
multipathd.  The good news is it would be very easy to move to another
process.

>>
> Precisely my idea.
> I'm fine with having a device-mapper D-Bus API, so that any application
> can retrieve the device-mapper layout via D-Bus.
> (It might as well be using sysfs directly, but who am I to argue here)
> Path events, however, out of necessity are instantiated within the
> kernel, and should be send out from the kernel via uevents.
> 
> D-Bus can be added on top of that, but multipathd should not generate
> path events for D-Bus.
> 

Where should D-Bus events be layered on top of the uevents if not inside
multipathd?

Would putting it inside multipathd be a reasonable first step?  We could
maintain the same public D-Bus API and move it to a different process.

Thanks,
Todd