All of lore.kernel.org
 help / color / mirror / Atom feed
From: Roman Bolshakov <r.bolshakov@yadro.com>
To: linux-scsi@vger.kernel.org, target-devel@vger.kernel.org
Cc: linux@yadro.com, Roman Bolshakov <r.bolshakov@yadro.com>,
	Quinn Tran <qutran@marvell.com>,
	Bart Van Assche <bvanassche@acm.org>,
	Thomas Abraham <tabraham@suse.com>,
	stable@vger.kernel.org, Himanshu Madhani <hmadhani@marvell.com>
Subject: [PATCH v2 01/15] scsi: qla2xxx: Ignore NULL pointer in tcm_qla2xxx_free_mcmd
Date: Wed, 20 Nov 2019 22:27:09 +0000	[thread overview]
Message-ID: <20191120222723.27779-2-r.bolshakov@yadro.com> (raw)
In-Reply-To: <20191120222723.27779-1-r.bolshakov@yadro.com>

If ABTS cannot be completed in target mode, the driver attempts to free
related management command and crashes:

  NIP [d000000019181ee8] tcm_qla2xxx_free_mcmd+0x40/0x80 [tcm_qla2xxx]
  LR [d00000001dc1e6f8] qlt_response_pkt+0x190/0xa10 [qla2xxx]
  Call Trace:
  [c000003fff27bb50] [c000003fff27bc10] 0xc000003fff27bc10 (unreliable)
  [c000003fff27bb70] [d00000001dc1e6f8] qlt_response_pkt+0x190/0xa10 [qla2xxx]
  [c000003fff27bc10] [d00000001dbc2be0] qla24xx_process_response_queue+0x5d8/0xbd0 [qla2xxx]
  [c000003fff27bd50] [d00000001dbc632c] qla24xx_msix_rsp_q+0x64/0x150 [qla2xxx]
  [c000003fff27bde0] [c000000000187200] __handle_irq_event_percpu+0x90/0x310
  [c000003fff27bea0] [c0000000001874b8] handle_irq_event_percpu+0x38/0x90
  [c000003fff27bee0] [c000000000187574] handle_irq_event+0x64/0xb0
  [c000003fff27bf10] [c00000000018cd38] handle_fasteoi_irq+0xe8/0x280
  [c000003fff27bf40] [c000000000185ccc] generic_handle_irq+0x4c/0x70
  [c000003fff27bf60] [c000000000016cec] __do_irq+0x7c/0x1d0
  [c000003fff27bf90] [c00000000002a530] call_do_irq+0x14/0x24
  [c00000207d2cba90] [c000000000016edc] do_IRQ+0x9c/0x130
  [c00000207d2cbae0] [c000000000008bf4] hardware_interrupt_common+0x114/0x120
  --- interrupt: 501 at arch_local_irq_restore+0x74/0x90
      LR = arch_local_irq_restore+0x74/0x90
  [c00000207d2cbdd0] [c0000000001c64fc] tick_broadcast_oneshot_control+0x4c/0x60 (unreliable)
  [c00000207d2cbdf0] [c0000000007ac840] cpuidle_enter_state+0xf0/0x450
  [c00000207d2cbe50] [c00000000016b81c] call_cpuidle+0x4c/0x90
  [c00000207d2cbe70] [c00000000016bc30] do_idle+0x2b0/0x330
  [c00000207d2cbec0] [c00000000016beec] cpu_startup_entry+0x3c/0x50
  [c00000207d2cbef0] [c00000000004a06c] start_secondary+0x63c/0x670
  [c00000207d2cbf90] [c00000000000aa6c] start_secondary_prolog+0x10/0x14

The crash can be triggered by ACL deletion when there's active I/O.

During ACL deletion, qla2xxx performs implicit LOGO that's invisible for
the initiator. Only the driver and firmware are aware of the logout.
Therefore the initiator continues to send SCSI commands and the target
always responds with SAM STATUS BUSY as it can't find the session.

The command times out after a while and initiator invokes ABORT TASK TMF
for the command. The TMF is mapped to ABTS-LS in FCP. The target can't
find session for S_ID originating ABTS-LS so it never allocates mcmd.
And since N_Port handle was deleted after LOGO, it is no longer valid
and ABTS Response IOCB is returned from firmware with status 31. Then
free_mcmd is invoked on NULL pointer and the kernel crashes.

[ 7734.578642] qla2xxx [0000:00:0c.0]-e837:6: ABTS_RECV_24XX: instance 0
[ 7734.578644] qla2xxx [0000:00:0c.0]-f811:6: qla_target(0): task abort (s_id=1:2:0, tag\x1209504, param=0)
[ 7734.578645] find_sess_by_s_id: 0x010200
[ 7734.578645] Unable to locate s_id: 0x010200
[ 7734.578646] qla2xxx [0000:00:0c.0]-f812:6: qla_target(0): task abort for non-existent session
[ 7734.578648] qla2xxx [0000:00:0c.0]-e806:6: Sending task mgmt ABTS response (ha└000000d5819000, atio└000000d3fd4700, status=4
[ 7734.578730] qla2xxx [0000:00:0c.0]-e838:6: ABTS_RESP_24XX: compl_status 31
[ 7734.578732] qla2xxx [0000:00:0c.0]-e863:6: qla_target(0): ABTS_RESP_24XX failed 31 (subcode 19:a)
[ 7734.578740] Unable to handle kernel paging request for data at address 0x00000200

Fixes: 6b0431d6fa20b ("scsi: qla2xxx: Fix out of order Termination and ABTS response")
Cc: Quinn Tran <qutran@marvell.com>
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Thomas Abraham <tabraham@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Acked-by: Himanshu Madhani <hmadhani@marvell.com>
---
 drivers/scsi/qla2xxx/tcm_qla2xxx.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
index 042a24314edc..bab2073c1f72 100644
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
@@ -246,6 +246,8 @@ static void tcm_qla2xxx_complete_mcmd(struct work_struct *work)
  */
 static void tcm_qla2xxx_free_mcmd(struct qla_tgt_mgmt_cmd *mcmd)
 {
+	if (!mcmd)
+		return;
 	INIT_WORK(&mcmd->free_work, tcm_qla2xxx_complete_mcmd);
 	queue_work(tcm_qla2xxx_free_wq, &mcmd->free_work);
 }
-- 
2.24.0

WARNING: multiple messages have this Message-ID (diff)
From: Roman Bolshakov <r.bolshakov@yadro.com>
To: <linux-scsi@vger.kernel.org>, <target-devel@vger.kernel.org>
Cc: <linux@yadro.com>, Roman Bolshakov <r.bolshakov@yadro.com>,
	Quinn Tran <qutran@marvell.com>,
	Bart Van Assche <bvanassche@acm.org>,
	Thomas Abraham <tabraham@suse.com>, <stable@vger.kernel.org>,
	Himanshu Madhani <hmadhani@marvell.com>
Subject: [PATCH v2 01/15] scsi: qla2xxx: Ignore NULL pointer in tcm_qla2xxx_free_mcmd
Date: Thu, 21 Nov 2019 01:27:09 +0300	[thread overview]
Message-ID: <20191120222723.27779-2-r.bolshakov@yadro.com> (raw)
In-Reply-To: <20191120222723.27779-1-r.bolshakov@yadro.com>

If ABTS cannot be completed in target mode, the driver attempts to free
related management command and crashes:

  NIP [d000000019181ee8] tcm_qla2xxx_free_mcmd+0x40/0x80 [tcm_qla2xxx]
  LR [d00000001dc1e6f8] qlt_response_pkt+0x190/0xa10 [qla2xxx]
  Call Trace:
  [c000003fff27bb50] [c000003fff27bc10] 0xc000003fff27bc10 (unreliable)
  [c000003fff27bb70] [d00000001dc1e6f8] qlt_response_pkt+0x190/0xa10 [qla2xxx]
  [c000003fff27bc10] [d00000001dbc2be0] qla24xx_process_response_queue+0x5d8/0xbd0 [qla2xxx]
  [c000003fff27bd50] [d00000001dbc632c] qla24xx_msix_rsp_q+0x64/0x150 [qla2xxx]
  [c000003fff27bde0] [c000000000187200] __handle_irq_event_percpu+0x90/0x310
  [c000003fff27bea0] [c0000000001874b8] handle_irq_event_percpu+0x38/0x90
  [c000003fff27bee0] [c000000000187574] handle_irq_event+0x64/0xb0
  [c000003fff27bf10] [c00000000018cd38] handle_fasteoi_irq+0xe8/0x280
  [c000003fff27bf40] [c000000000185ccc] generic_handle_irq+0x4c/0x70
  [c000003fff27bf60] [c000000000016cec] __do_irq+0x7c/0x1d0
  [c000003fff27bf90] [c00000000002a530] call_do_irq+0x14/0x24
  [c00000207d2cba90] [c000000000016edc] do_IRQ+0x9c/0x130
  [c00000207d2cbae0] [c000000000008bf4] hardware_interrupt_common+0x114/0x120
  --- interrupt: 501 at arch_local_irq_restore+0x74/0x90
      LR = arch_local_irq_restore+0x74/0x90
  [c00000207d2cbdd0] [c0000000001c64fc] tick_broadcast_oneshot_control+0x4c/0x60 (unreliable)
  [c00000207d2cbdf0] [c0000000007ac840] cpuidle_enter_state+0xf0/0x450
  [c00000207d2cbe50] [c00000000016b81c] call_cpuidle+0x4c/0x90
  [c00000207d2cbe70] [c00000000016bc30] do_idle+0x2b0/0x330
  [c00000207d2cbec0] [c00000000016beec] cpu_startup_entry+0x3c/0x50
  [c00000207d2cbef0] [c00000000004a06c] start_secondary+0x63c/0x670
  [c00000207d2cbf90] [c00000000000aa6c] start_secondary_prolog+0x10/0x14

The crash can be triggered by ACL deletion when there's active I/O.

During ACL deletion, qla2xxx performs implicit LOGO that's invisible for
the initiator. Only the driver and firmware are aware of the logout.
Therefore the initiator continues to send SCSI commands and the target
always responds with SAM STATUS BUSY as it can't find the session.

The command times out after a while and initiator invokes ABORT TASK TMF
for the command. The TMF is mapped to ABTS-LS in FCP. The target can't
find session for S_ID originating ABTS-LS so it never allocates mcmd.
And since N_Port handle was deleted after LOGO, it is no longer valid
and ABTS Response IOCB is returned from firmware with status 31. Then
free_mcmd is invoked on NULL pointer and the kernel crashes.

[ 7734.578642] qla2xxx [0000:00:0c.0]-e837:6: ABTS_RECV_24XX: instance 0
[ 7734.578644] qla2xxx [0000:00:0c.0]-f811:6: qla_target(0): task abort (s_id=1:2:0, tag=1209504, param=0)
[ 7734.578645] find_sess_by_s_id: 0x010200
[ 7734.578645] Unable to locate s_id: 0x010200
[ 7734.578646] qla2xxx [0000:00:0c.0]-f812:6: qla_target(0): task abort for non-existent session
[ 7734.578648] qla2xxx [0000:00:0c.0]-e806:6: Sending task mgmt ABTS response (ha=c0000000d5819000, atio=c0000000d3fd4700, status=4
[ 7734.578730] qla2xxx [0000:00:0c.0]-e838:6: ABTS_RESP_24XX: compl_status 31
[ 7734.578732] qla2xxx [0000:00:0c.0]-e863:6: qla_target(0): ABTS_RESP_24XX failed 31 (subcode 19:a)
[ 7734.578740] Unable to handle kernel paging request for data at address 0x00000200

Fixes: 6b0431d6fa20b ("scsi: qla2xxx: Fix out of order Termination and ABTS response")
Cc: Quinn Tran <qutran@marvell.com>
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Thomas Abraham <tabraham@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Acked-by: Himanshu Madhani <hmadhani@marvell.com>
---
 drivers/scsi/qla2xxx/tcm_qla2xxx.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
index 042a24314edc..bab2073c1f72 100644
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
@@ -246,6 +246,8 @@ static void tcm_qla2xxx_complete_mcmd(struct work_struct *work)
  */
 static void tcm_qla2xxx_free_mcmd(struct qla_tgt_mgmt_cmd *mcmd)
 {
+	if (!mcmd)
+		return;
 	INIT_WORK(&mcmd->free_work, tcm_qla2xxx_complete_mcmd);
 	queue_work(tcm_qla2xxx_free_wq, &mcmd->free_work);
 }
-- 
2.24.0


  reply	other threads:[~2019-11-20 22:27 UTC|newest]

Thread overview: 68+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-20 22:27 [PATCH v2 00/15] scsi: qla2xxx: Bug fixes Roman Bolshakov
2019-11-20 22:27 ` Roman Bolshakov
2019-11-20 22:27 ` Roman Bolshakov [this message]
2019-11-20 22:27   ` [PATCH v2 01/15] scsi: qla2xxx: Ignore NULL pointer in tcm_qla2xxx_free_mcmd Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 02/15] scsi: qla2xxx: Initialize free_work before flushing it Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 03/15] scsi: qla2xxx: Drop superfluous INIT_WORK of del_work Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 04/15] scsi: qla2xxx: Change discovery state before PLOGI Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 05/15] scsi: qla2xxx: Allow PLOGI in target mode Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 22:52   ` [EXT] " Quinn Tran
2019-11-21 16:38     ` Himanshu Madhani
2019-11-21 16:38       ` Himanshu Madhani
2019-11-20 22:27 ` [PATCH v2 06/15] scsi: qla2xxx: Don't call qlt_async_event twice Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-21 15:43   ` Himanshu Madhani
2019-11-21 15:43     ` Himanshu Madhani
2019-11-20 22:27 ` [PATCH v2 07/15] scsi: qla2xxx: Fix PLOGI payload and ELS IOCB dump length Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-21 16:39   ` [EXT] " Himanshu Madhani
2019-11-21 16:39     ` Himanshu Madhani
2019-11-20 22:27 ` [PATCH v2 08/15] scsi: qla2xxx: Configure local loop for N2N target Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 23:06   ` [EXT] " Quinn Tran
2019-11-21 16:39   ` Himanshu Madhani
2019-11-21 16:39     ` Himanshu Madhani
2019-11-20 22:27 ` [PATCH v2 09/15] scsi: qla2xxx: Send Notify ACK after N2N PLOGI Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 23:53   ` [EXT] " Quinn Tran
2019-11-21 16:40   ` Himanshu Madhani
2019-11-21 16:40     ` Himanshu Madhani
2019-11-20 22:27 ` [PATCH v2 10/15] scsi: qla2xxx: Don't defer relogin unconditonally Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-21  0:03   ` [EXT] " Quinn Tran
2019-11-20 22:27 ` [PATCH v2 11/15] scsi: qla2xxx: Ignore PORT UPDATE after N2N PLOGI Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 12/15] scsi: qla2xxx: Use explicit LOGO in target mode Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-23  4:58   ` kbuild test robot
2019-11-23  4:58     ` kbuild test robot
2019-11-23  4:58     ` kbuild test robot
2019-11-20 22:27 ` [PATCH v2 13/15] scsi: qla2xxx: Add async mode for qla24xx_els_dcmd_iocb Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-21 22:50   ` [EXT] " Quinn Tran
2019-11-22  5:04     ` Mark Harvey
2019-11-22  5:04       ` Mark Harvey
2019-11-22 17:00       ` Quinn Tran
2019-11-22 17:00         ` Quinn Tran
2019-11-24 19:15       ` Roman Bolshakov
2019-11-24 19:15         ` Roman Bolshakov
2019-11-20 22:27 ` [PATCH v2 14/15] scsi: qla2xxx: Add debug dump of LOGO payload and ELS IOCB Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-21 16:37   ` Himanshu Madhani
2019-11-21 16:37     ` Himanshu Madhani
2019-11-21 22:52   ` [EXT] " Quinn Tran
2019-11-20 22:27 ` [PATCH v2 15/15] scsi: qla2xxx: Handle ABTS according to FCP spec for logged out ports Roman Bolshakov
2019-11-20 22:27   ` Roman Bolshakov
2019-11-23  5:57   ` kbuild test robot
2019-11-23  5:57     ` kbuild test robot
2019-11-23  5:57     ` kbuild test robot
2019-11-22  9:14 ` [PATCH v2 00/15] scsi: qla2xxx: Bug fixes Hannes Reinecke
2019-11-22  9:14   ` Hannes Reinecke
2019-11-22 21:36   ` Martin Wilck
2019-11-22 21:36     ` Martin Wilck
2019-11-24 18:31     ` Roman Bolshakov
2019-11-24 18:31       ` Roman Bolshakov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191120222723.27779-2-r.bolshakov@yadro.com \
    --to=r.bolshakov@yadro.com \
    --cc=bvanassche@acm.org \
    --cc=hmadhani@marvell.com \
    --cc=linux-scsi@vger.kernel.org \
    --cc=linux@yadro.com \
    --cc=qutran@marvell.com \
    --cc=stable@vger.kernel.org \
    --cc=tabraham@suse.com \
    --cc=target-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.