All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
@ 2018-02-01 22:16 Mauricio Faria de Oliveira
  2018-02-01 22:27 ` test results Mauricio Faria de Oliveira
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Mauricio Faria de Oliveira @ 2018-02-01 22:16 UTC (permalink / raw)
  To: linux-scsi, Bart.VanAssche
  Cc: sathya.prakash, chaitra.basappa, suganath-prabu.subramani, jejb,
	martin.petersen, dougmill

This patch adds checks for 'ioc->remove_host' in the SCSI error
handlers, so not to access pointers/resources potentially freed
in the PCI shutdown/module unload path.  The error handlers may
be invoked after shutdown/unload, depending on other components.

This problem was observed with kexec on a system with a mpt3sas
based adapter and an infiniband adapter which takes long enough
to shutdown:

The mpt3sas driver finished shutting down / disabled interrupt
handling, thus some commands have not finished and timed out.

Since the system was still running (waiting for the infiniband
adapter to shutdown), the scsi error handler for task abort of
mpt3sas was invoked, and hit an oops -- either in scsih_abort()
because 'ioc->scsi_lookup' was NULL (without commit dbec4c9040ed
("scsi: mpt3sas: lockless command submission")), or later up in
scsih_host_reset() (with or without that commit), because it
eventually called mpt3sas_base_get_iocstate().

After that commit, the oops in scsih_abort() does not occur
anymore (_scsih_scsi_lookup_find_by_scmd() is no longer called),
but that commit is too big and out of the scope of linux-stable,
where this patch might help, so still go for the changes.

Also, this might help to prevent similar errors in the future,
in case code changes and possibly tries to access freed stuff.

Note the fix in scsih_host_reset() is still important anyway.

Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
v2:
 - rebase on top of mkp/scsi.git's fixes branch
 - insert check for 'ioc->remove_host' in existing
   checks which already set DID_NO_CONNECT instead
   of duplicating that code. (helps with backports)
 - update commit message about that commit.

 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 74fca18..5ab3caf 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2835,7 +2835,8 @@ int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2898,7 +2899,8 @@ int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2961,7 +2963,8 @@ int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		starget_printk(KERN_INFO, starget, "target been deleted! scmd(%p)\n",
 			scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -3019,7 +3022,7 @@ int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	    ioc->name, scmd);
 	scsi_print_command(scmd);
 
-	if (ioc->is_driver_loading) {
+	if (ioc->is_driver_loading || ioc->remove_host) {
 		pr_info(MPT3SAS_FMT "Blocking the host reset\n",
 		    ioc->name);
 		r = FAILED;
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* test results
  2018-02-01 22:16 [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Mauricio Faria de Oliveira
@ 2018-02-01 22:27 ` Mauricio Faria de Oliveira
  2018-02-09 23:22 ` [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Martin K. Petersen
  2018-02-12  6:28 ` Sreekanth Reddy
  2 siblings, 0 replies; 7+ messages in thread
From: Mauricio Faria de Oliveira @ 2018-02-01 22:27 UTC (permalink / raw)
  To: linux-scsi, Bart.VanAssche
  Cc: sathya.prakash, chaitra.basappa, suganath-prabu.subramani, jejb,
	martin.petersen, dougmill

The test-case results with PATCH v2.

scsih_abort()
=============

Without patch:

    [  362.669743] setting logging_level(0x10000000)
    [  362.705074] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fd4f2bd40)
    [  363.956579] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  363.956844] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  363.957147] mpt3sas_cm0: sending diag reset !!
    [  365.073568] mpt3sas_cm0: diag reset: SUCCESS
    [  365.090316] mpt3sas_cm0: sleep on shutdown
    [  366.120209] mpt3sas_cm0: sleep on shutdown
    ...
    [  373.419954] sd 16:0:1:0: attempting task abort! scmd(c000001fd4f2bd40)
    ...
    [  373.420256] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.420300] sd 16:0:1:0: task abort: FAILED scmd(c000001fd4f2bd40)
    [  373.459961] sd 16:0:1:0: attempting device reset! scmd(c000001fd4f2bd40)
    ...
    [  373.460271] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.460315] sd 16:0:1:0: device reset: FAILED scmd(c000001fd4f2bd40)
    [  373.460362] scsi target16:0:1: attempting target reset! scmd(c000001fd4f2bd40)
    ...
    [  373.460628] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.460673] scsi target16:0:1: target reset: FAILED scmd(c000001fd4f2bd40)
    [  373.460720] mpt3sas_cm0: attempting host reset! scmd(c000001fd4f2bd40)
    [  373.460764] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  373.460821] Unable to handle kernel paging request for data at address 0xd0003800817d0000
    [  373.460876] Faulting instruction address: 0xd000000017b169b8
    [  373.460921] Oops: Kernel access of bad area, sig: 11 [#1]
    ...
    [  373.462028] NIP [d000000017b169b8] mpt3sas_base_get_iocstate+0x38/0xb0 [mpt3sas]
    [  373.462085] LR [d000000017b1a3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  373.462138] Call Trace:
    [  373.462160] [c000001fdf5f7ab0] [d000000017b1a3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  373.462225] [c000001fdf5f7b80] [d000000017b20f6c] scsih_host_reset+0x7c/0x100 [mpt3sas]
    [  373.462281] [c000001fdf5f7c00] [c00000000076f34c] scsi_try_host_reset+0x5c/0x140
    [  373.462335] [c000001fdf5f7c40] [c00000000077107c] scsi_eh_ready_devs+0x73c/0x960
    [  373.462390] [c000001fdf5f7d10] [c000000000772800] scsi_error_handler+0x4c0/0x4e0
    [  373.462444] [c000001fdf5f7dc0] [c000000000107ed4] kthread+0x164/0x1b0
    [  373.462490] [c000001fdf5f7e30] [c00000000000b5a0] ret_from_kernel_thread+0x5c/0xbc
    [  373.468473] Instruction dump:

With patch:

    [  332.994654] setting logging_level(0x10000000)
    [  333.024246] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000003c97348140)
    [  334.232041] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  334.232324] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  334.232598] mpt3sas_cm0: sending diag reset !!
    [  335.352533] mpt3sas_cm0: diag reset: SUCCESS
    [  335.372075] mpt3sas_cm0: sleep on shutdown
    [  336.440544] mpt3sas_cm0: sleep on shutdown
    ...
    [  343.100179] sd 16:0:1:0: attempting task abort! scmd(c000003c97348140)
    ...
    [  343.100488] sd 16:0:1:0: device been deleted! scmd(c000003c97348140)
    [  343.100532] sd 16:0:1:0: task abort: SUCCESS scmd(c000003c97348140)
    [  343.140185] sd 16:0:1:0: [sdf] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [  343.140275] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  343.140338] print_req_error: I/O error, dev sdf, sector 0


scsih_dev/target/host_reset()
===========================

Without patch:

    [  101.077946] setting logging_level(0x11000000)
    [  101.110029] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fb7a2dd40)
    [  102.356108] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  102.356396] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  102.356647] mpt3sas_cm0: sending diag reset !!
    [  103.476380] mpt3sas_cm0: diag reset: SUCCESS
    [  103.492696] mpt3sas_cm0: sleep on shutdown
    [  104.512914] mpt3sas_cm0: sleep on shutdown
    ...
    [  111.732912] sd 16:0:1:0: attempting task abort! scmd(c000001fb7a2dd40)
    ...
    [  111.733202] mpt3sas_cm0: fail task abort scmd(c000001fb7a2dd40)
    [  111.733246] sd 16:0:1:0: task abort: FAILED scmd(c000001fb7a2dd40)
    [  111.772919] sd 16:0:1:0: attempting device reset! scmd(c000001fb7a2dd40)
    ...
    [  111.773177] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  111.773222] sd 16:0:1:0: device reset: FAILED scmd(c000001fb7a2dd40)
    [  111.773266] scsi target16:0:1: attempting target reset! scmd(c000001fb7a2dd40)
    ...
    [  111.773528] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  111.773574] scsi target16:0:1: target reset: FAILED scmd(c000001fb7a2dd40)
    [  111.773617] mpt3sas_cm0: attempting host reset! scmd(c000001fb7a2dd40)
    [  111.773662] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  111.773721] Unable to handle kernel paging request for data at address 0xd000380081790000
    [  111.773773] Faulting instruction address: 0xd0000000179d69b8
    [  111.773818] Oops: Kernel access of bad area, sig: 11 [#1]
    ...
    [  111.774925] NIP [d0000000179d69b8] mpt3sas_base_get_iocstate+0x38/0xb0 [mpt3sas]
    [  111.774980] LR [d0000000179da3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  111.775032] Call Trace:
    [  111.775054] [c000001fd995bab0] [d0000000179da3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  111.775118] [c000001fd995bb80] [d0000000179e0f6c] scsih_host_reset+0x7c/0x100 [mpt3sas]
    [  111.775173] [c000001fd995bc00] [c00000000076f34c] scsi_try_host_reset+0x5c/0x140
    [  111.775227] [c000001fd995bc40] [c00000000077107c] scsi_eh_ready_devs+0x73c/0x960
    [  111.775280] [c000001fd995bd10] [c000000000772800] scsi_error_handler+0x4c0/0x4e0
    [  111.775333] [c000001fd995bdc0] [c000000000107ed4] kthread+0x164/0x1b0
    [  111.775378] [c000001fd995be30] [c00000000000b5a0] ret_from_kernel_thread+0x5c/0xbc
    [  111.778393] Instruction dump:

With patch:

    [   56.316855] setting logging_level(0x11000000)
    [   56.347607] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fc3afb540)
    [   57.604052] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [   57.604319] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [   57.604615] mpt3sas_cm0: sending diag reset !!
    [   58.725154] mpt3sas_cm0: diag reset: SUCCESS
    [   58.741245] mpt3sas_cm0: sleep on shutdown
    [   59.803191] mpt3sas_cm0: sleep on shutdown
    ...
    [   67.103187] sd 16:0:1:0: attempting task abort! scmd(c000001fc3afb540)
    ...
    [   67.103521] mpt3sas_cm0: fail task abort scmd(c000001fc3afb540)
    [   67.103569] sd 16:0:1:0: task abort: FAILED scmd(c000001fc3afb540)
    [   67.143193] sd 16:0:1:0: attempting device reset! scmd(c000001fc3afb540)
    ...
    [   67.143514] sd 16:0:1:0: device been deleted! scmd(c000001fc3afb540)
    [   67.143564] sd 16:0:1:0: device reset: SUCCESS scmd(c000001fc3afb540)
    [   67.143613] scsi target16:0:1: attempting target reset! scmd(c000001fc3afb540)
    ...
    [   67.143897] scsi target16:0:1: target been deleted! scmd(c000001fc3afb540)
    [   67.143944] scsi target16:0:1: target reset: SUCCESS scmd(c000001fc3afb540)
    [   67.143992] mpt3sas_cm0: attempting host reset! scmd(c000001fc3afb540)
    [   67.144040] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [   67.144096] mpt3sas_cm0: Blocking the host reset
    [   67.144133] mpt3sas_cm0: host reset: FAILED scmd(c000001fc3afb540)
    [   67.144179] sd 16:0:1:0: Device offlined - not ready after error recovery
    [   67.144228] sd 16:0:1:0: [sdf] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [   67.144295] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [   67.144351] print_req_error: I/O error, dev sdf, sector 0

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
  2018-02-01 22:16 [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Mauricio Faria de Oliveira
  2018-02-01 22:27 ` test results Mauricio Faria de Oliveira
@ 2018-02-09 23:22 ` Martin K. Petersen
  2018-02-12  6:28 ` Sreekanth Reddy
  2 siblings, 0 replies; 7+ messages in thread
From: Martin K. Petersen @ 2018-02-09 23:22 UTC (permalink / raw)
  To: Mauricio Faria de Oliveira
  Cc: linux-scsi, Bart.VanAssche, sathya.prakash, chaitra.basappa,
	suganath-prabu.subramani, jejb, martin.petersen, dougmill


> This patch adds checks for 'ioc->remove_host' in the SCSI error
> handlers, so not to access pointers/resources potentially freed
> in the PCI shutdown/module unload path.  The error handlers may
> be invoked after shutdown/unload, depending on other components.

Broadcom folks: Please review!

-- 
Martin K. Petersen	Oracle Linux Engineering

^ permalink raw reply	[flat|nested] 7+ messages in thread

* RE: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
  2018-02-01 22:16 [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Mauricio Faria de Oliveira
  2018-02-01 22:27 ` test results Mauricio Faria de Oliveira
  2018-02-09 23:22 ` [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Martin K. Petersen
@ 2018-02-12  6:28 ` Sreekanth Reddy
  2018-02-14 15:35   ` Mauricio Faria de Oliveira
  2 siblings, 1 reply; 7+ messages in thread
From: Sreekanth Reddy @ 2018-02-12  6:28 UTC (permalink / raw)
  To: Mauricio Faria de Oliveira, linux-scsi, Bart.VanAssche
  Cc: Sathya Prakash Veerichetty, Chaitra Basappa,
	Suganath Prabu Subramani, jejb, martin.petersen, dougmill

Mauricio,

Instead of returning the scmd with DID_NO_CONNECT in TM path, can we wait
for some time (10 seconds) in shutdown/unload path for the outstanding
commands to complete and even then the scmds are outstanding then return
all the outstanding scmds with DID_NO_CONNECT in the shutdown/unload path
itself as shown below,

----
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c
b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 13d6e4e..f62ce50 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -6294,14 +6294,14 @@ _base_reset_handler(struct MPT3SAS_ADAPTER *ioc,
int reset_phase)
 }

 /**
- * _wait_for_commands_to_complete - reset controller
+ * mpt3sas_wait_for_commands_to_complete - reset controller
  * @ioc: Pointer to MPT_ADAPTER structure
  *
  * This function is waiting 10s for all pending commands to complete
  * prior to putting controller in reset.
  */
-static void
-_wait_for_commands_to_complete(struct MPT3SAS_ADAPTER *ioc)
+void
+mpt3sas_wait_for_commands_to_complete(struct MPT3SAS_ADAPTER *ioc)
 {
 	u32 ioc_state;

@@ -6374,7 +6374,7 @@ mpt3sas_base_hard_reset_handler(struct
MPT3SAS_ADAPTER *ioc,
 			is_fault = 1;
 	}
 	_base_reset_handler(ioc, MPT3_IOC_PRE_RESET);
-	_wait_for_commands_to_complete(ioc);
+	mpt3sas_wait_for_commands_to_complete(ioc);
 	_base_mask_interrupts(ioc);
 	r = _base_make_ioc_ready(ioc, type);
 	if (r)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h
b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 789bc42..99ccf83 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1433,6 +1433,9 @@ void mpt3sas_base_update_missing_delay(struct
MPT3SAS_ADAPTER *ioc,

 int mpt3sas_port_enable(struct MPT3SAS_ADAPTER *ioc);

+void
+mpt3sas_wait_for_commands_to_complete(struct MPT3SAS_ADAPTER *ioc);
+

 /* scsih shared API */
 struct scsi_cmnd *mpt3sas_scsih_scsi_lookup_get(struct MPT3SAS_ADAPTER
*ioc,
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 74fca18..458709e 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4453,7 +4453,7 @@ _scsih_flush_running_cmds(struct MPT3SAS_ADAPTER
*ioc)
 		st = scsi_cmd_priv(scmd);
 		mpt3sas_base_clear_st(ioc, st);
 		scsi_dma_unmap(scmd);
-		if (ioc->pci_error_recovery)
+		if (ioc->pci_error_recovery || ioc->remove_host)
 			scmd->result = DID_NO_CONNECT << 16;
 		else
 			scmd->result = DID_RESET << 16;
@@ -9739,6 +9739,10 @@ static void scsih_remove(struct pci_dev *pdev)
 	unsigned long flags;

 	ioc->remove_host = 1;
+
+	mpt3sas_wait_for_commands_to_complete(ioc);
+	_scsih_flush_running_cmds(ioc);
+
 	_scsih_fw_event_cleanup_queue(ioc);

 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
@@ -9815,6 +9819,10 @@ scsih_shutdown(struct pci_dev *pdev)
 	unsigned long flags;

 	ioc->remove_host = 1;
+
+	mpt3sas_wait_for_commands_to_complete(ioc);
+	_scsih_flush_running_cmds(ioc);
+
 	_scsih_fw_event_cleanup_queue(ioc);

 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
---

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org
[mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Mauricio Faria de
Oliveira
Sent: Friday, February 2, 2018 3:46 AM
To: linux-scsi@vger.kernel.org; Bart.VanAssche@wdc.com
Cc: sathya.prakash@broadcom.com; chaitra.basappa@broadcom.com;
suganath-prabu.subramani@broadcom.com; jejb@linux.vnet.ibm.com;
martin.petersen@oracle.com; dougmill@linux.vnet.ibm.com
Subject: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after
shutdown/unload

This patch adds checks for 'ioc->remove_host' in the SCSI error handlers,
so not to access pointers/resources potentially freed in the PCI
shutdown/module unload path.  The error handlers may be invoked after
shutdown/unload, depending on other components.

This problem was observed with kexec on a system with a mpt3sas based
adapter and an infiniband adapter which takes long enough to shutdown:

The mpt3sas driver finished shutting down / disabled interrupt handling,
thus some commands have not finished and timed out.

Since the system was still running (waiting for the infiniband adapter to
shutdown), the scsi error handler for task abort of mpt3sas was invoked,
and hit an oops -- either in scsih_abort() because 'ioc->scsi_lookup' was
NULL (without commit dbec4c9040ed
("scsi: mpt3sas: lockless command submission")), or later up in
scsih_host_reset() (with or without that commit), because it eventually
called mpt3sas_base_get_iocstate().

After that commit, the oops in scsih_abort() does not occur anymore
(_scsih_scsi_lookup_find_by_scmd() is no longer called), but that commit
is too big and out of the scope of linux-stable, where this patch might
help, so still go for the changes.

Also, this might help to prevent similar errors in the future, in case
code changes and possibly tries to access freed stuff.

Note the fix in scsih_host_reset() is still important anyway.

Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
v2:
 - rebase on top of mkp/scsi.git's fixes branch
 - insert check for 'ioc->remove_host' in existing
   checks which already set DID_NO_CONNECT instead
   of duplicating that code. (helps with backports)
 - update commit message about that commit.

 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 74fca18..5ab3caf 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2835,7 +2835,8 @@ int mpt3sas_scsih_issue_locked_tm(struct
MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);

 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2898,7 +2899,8 @@ int mpt3sas_scsih_issue_locked_tm(struct
MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);

 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2961,7 +2963,8 @@ int mpt3sas_scsih_issue_locked_tm(struct
MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);

 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		starget_printk(KERN_INFO, starget, "target been deleted!
scmd(%p)\n",
 			scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -3019,7 +3022,7 @@ int mpt3sas_scsih_issue_locked_tm(struct
MPT3SAS_ADAPTER *ioc, u16 handle,
 	    ioc->name, scmd);
 	scsi_print_command(scmd);

-	if (ioc->is_driver_loading) {
+	if (ioc->is_driver_loading || ioc->remove_host) {
 		pr_info(MPT3SAS_FMT "Blocking the host reset\n",
 		    ioc->name);
 		r = FAILED;
--
1.8.3.1

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
  2018-02-12  6:28 ` Sreekanth Reddy
@ 2018-02-14 15:35   ` Mauricio Faria de Oliveira
  2018-02-15  5:48     ` Sreekanth Reddy
  0 siblings, 1 reply; 7+ messages in thread
From: Mauricio Faria de Oliveira @ 2018-02-14 15:35 UTC (permalink / raw)
  To: Sreekanth Reddy, linux-scsi, Bart.VanAssche
  Cc: Sathya Prakash Veerichetty, Chaitra Basappa,
	Suganath Prabu Subramani, jejb, martin.petersen, dougmill

Hi Sreenkanth,

Thanks for reviewing.

On 02/12/2018 04:28 AM, Sreekanth Reddy wrote:
> Instead of returning the scmd with DID_NO_CONNECT in TM path, can we wait
> for some time (10 seconds) in shutdown/unload path for the outstanding
> commands to complete and even then [if] the scmds are outstanding then return
> all the outstanding scmds with DID_NO_CONNECT in the shutdown/unload path
> itself as shown below,

Apparently there is a problem with that.

That is not enough for TM commands; it's only for SCSI IO commands.

The TM commands use the high-priority queue, and SCSI IO commands use
the SCSI IO queue.  But this patch only waits for and return commands
in the latter:

 > +	mpt3sas_wait_for_commands_to_complete(ioc);
 > +	_scsih_flush_running_cmds(ioc);

They only loop up until 'ioc->scsiio_depth', but TM commands get SMIDs
above that [see mpt3sas_base_free_smid() and mpt3sas_scsih_issue_tm()].

So, there's an exposure for abort/reset/other TM commands. For example,
the SCSI IO commands finish quickly in wait_for_commands_to_complete(),
then the shutdown function continues, disables interrupts, and release
pointers/memory.  Then, an outstanding abort/reset command is finished,
its completion interrupt is not serviced, and the SCSI EH for it kicks
in/continues, and so it escalates up, and hit that oops in host reset.

Is there any problem with the patch that I proposed?

It seems okay to have another check in error path (not hot/performance
path), and that check is already used in many other points in the code,
for the same reasons (exit early before the code attempts to use stuff
that might be released).

Thanks again,

-- 
Mauricio Faria de Oliveira
IBM Linux Technology Center

^ permalink raw reply	[flat|nested] 7+ messages in thread

* RE: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
  2018-02-14 15:35   ` Mauricio Faria de Oliveira
@ 2018-02-15  5:48     ` Sreekanth Reddy
  2018-02-15 12:10       ` Mauricio Faria de Oliveira
  0 siblings, 1 reply; 7+ messages in thread
From: Sreekanth Reddy @ 2018-02-15  5:48 UTC (permalink / raw)
  To: Mauricio Faria de Oliveira, linux-scsi, Bart.VanAssche
  Cc: Sathya Prakash Veerichetty, Chaitra Basappa,
	Suganath Prabu Subramani, jejb, martin.petersen, dougmill

Mauricio,

During the shutdown time, I don't want the outstanding IOs to timeout due to
disabling of interrupts and go the TM path. So I wanted to clear out all the
Outstanding IOs in the shutdown path itself instead of clearing them in TM
path.

But if there are already some TMs are outstanding while initiating the
shutdown operation then your patch looks good to me.

May be can you include my set of changes along with your solution?

Thanks,
Sreekanth

-----Original Message-----
From: Mauricio Faria de Oliveira [mailto:mauricfo@linux.vnet.ibm.com]
Sent: Wednesday, February 14, 2018 9:05 PM
To: Sreekanth Reddy; linux-scsi@vger.kernel.org; Bart.VanAssche@wdc.com
Cc: Sathya Prakash Veerichetty; Chaitra Basappa; Suganath Prabu Subramani;
jejb@linux.vnet.ibm.com; martin.petersen@oracle.com;
dougmill@linux.vnet.ibm.com
Subject: Re: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after
shutdown/unload

Hi Sreenkanth,

Thanks for reviewing.

On 02/12/2018 04:28 AM, Sreekanth Reddy wrote:
> Instead of returning the scmd with DID_NO_CONNECT in TM path, can we
> wait for some time (10 seconds) in shutdown/unload path for the
> outstanding commands to complete and even then [if] the scmds are
> outstanding then return all the outstanding scmds with DID_NO_CONNECT
> in the shutdown/unload path itself as shown below,

Apparently there is a problem with that.

That is not enough for TM commands; it's only for SCSI IO commands.

The TM commands use the high-priority queue, and SCSI IO commands use the
SCSI IO queue.  But this patch only waits for and return commands in the
latter:

 > +	mpt3sas_wait_for_commands_to_complete(ioc);
 > +	_scsih_flush_running_cmds(ioc);

They only loop up until 'ioc->scsiio_depth', but TM commands get SMIDs above
that [see mpt3sas_base_free_smid() and mpt3sas_scsih_issue_tm()].

So, there's an exposure for abort/reset/other TM commands. For example, the
SCSI IO commands finish quickly in wait_for_commands_to_complete(), then the
shutdown function continues, disables interrupts, and release
pointers/memory.  Then, an outstanding abort/reset command is finished, its
completion interrupt is not serviced, and the SCSI EH for it kicks
in/continues, and so it escalates up, and hit that oops in host reset.

Is there any problem with the patch that I proposed?

It seems okay to have another check in error path (not hot/performance
path), and that check is already used in many other points in the code, for
the same reasons (exit early before the code attempts to use stuff that
might be released).

Thanks again,

--
Mauricio Faria de Oliveira
IBM Linux Technology Center

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload
  2018-02-15  5:48     ` Sreekanth Reddy
@ 2018-02-15 12:10       ` Mauricio Faria de Oliveira
  0 siblings, 0 replies; 7+ messages in thread
From: Mauricio Faria de Oliveira @ 2018-02-15 12:10 UTC (permalink / raw)
  To: Sreekanth Reddy, linux-scsi, Bart.VanAssche
  Cc: Sathya Prakash Veerichetty, Chaitra Basappa,
	Suganath Prabu Subramani, jejb, martin.petersen, dougmill

Hi Sreekanth,

On 02/15/2018 03:48 AM, Sreekanth Reddy wrote:
> During the shutdown time, I don't want the outstanding IOs to timeout due to
> disabling of interrupts and go the TM path. So I wanted to clear out all the
> Outstanding IOs in the shutdown path itself instead of clearing them in TM
> path.

Ah. I see. I didn't realize that. Good catch.

> But if there are already some TMs are outstanding while initiating the
> shutdown operation then your patch looks good to me.

Ok. Thanks for the review.

> May be can you include my set of changes along with your solution?

Sure. I'll send a v3 with both parts and due credit.

cheers,
Mauricio

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2018-02-15 12:10 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-01 22:16 [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Mauricio Faria de Oliveira
2018-02-01 22:27 ` test results Mauricio Faria de Oliveira
2018-02-09 23:22 ` [PATCH v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload Martin K. Petersen
2018-02-12  6:28 ` Sreekanth Reddy
2018-02-14 15:35   ` Mauricio Faria de Oliveira
2018-02-15  5:48     ` Sreekanth Reddy
2018-02-15 12:10       ` Mauricio Faria de Oliveira

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.