All of lore.kernel.org
 help / color / mirror / Atom feed
From: Maurizio Lombardi <mlombard@redhat.com>
To: martin.petersen@oracle.com
Cc: linux-scsi@vger.kernel.org, target-devel@vger.kernel.org,
	bvanassche@acm.org, michael.christie@oracle.com
Subject: [PATCH 2/2] target: iscsi: fix a race condition when aborting a task
Date: Wed, 07 Oct 2020 14:53:26 +0000	[thread overview]
Message-ID: <20201007145326.56850-3-mlombard@redhat.com> (raw)
In-Reply-To: <20201007145326.56850-1-mlombard@redhat.com>

The iscsit_release_commands_from_conn() function does the following
operations:

1) locks the cmd_lock spinlock
2) Scans the list of commands and sets the CMD_T_FABRIC_STOP flag
3) Releases the cmd_lock spinlock
4) Rescans the list again and clears the i_conn_node link of each command

If an abort task timer is fired between 3) and 4), it will find
the CMD_T_FABRIC_STOP flag set and won't call list_del_init();
therefore it may end up calling __iscsit_free_cmd() with a
non-empty i_conn_node list, thus triggering the warning.

Considering that:

- we expect list_del_init() to be executed by
  iscsit_release_commands_from_conn() when the CMD_T_FABRIC_STOP is set.
- iscsit_aborted_task() is the only function that calls __iscsit_free_cmd()
  directly, while all the other functions call iscsit_free_cmd().
- the warning in __iscsit_free_cmd() is a duplicate (the same warning
  can be found in iscsit_free_cmd().

We can fix the bug by simply removing the warning from __iscsit_free_cmd()

kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 1 PID: 21173 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
...
kernel: CPU: 1 PID: 21173 Comm: kworker/u8:3 Kdump: loaded Not tainted 3.10.0-1062.4.1.el7.x86_64 #1
kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
kernel: Call Trace:
kernel: [<ffffffff91d78ba4>] dump_stack+0x19/0x1b
kernel: [<ffffffff9169a958>] __warn+0xd8/0x100
kernel: [<ffffffff9169aa9d>] warn_slowpath_null+0x1d/0x20
kernel: [<ffffffffc0a6f69e>] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
kernel: [<ffffffffc0a70bc4>] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
kernel: [<ffffffffc0a7830a>] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
kernel: [<ffffffffc09fa516>] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
kernel: [<ffffffffc09f4d92>] core_tmr_abort_task+0x102/0x180 [target_core_mod]
kernel: [<ffffffffc09f7bb2>] target_tmr_work+0x152/0x170 [target_core_mod]
kernel: [<ffffffff916bd1df>] process_one_work+0x17f/0x440
kernel: [<ffffffff916be2f6>] worker_thread+0x126/0x3c0
kernel: [<ffffffff916be1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
kernel: [<ffffffff916c51b1>] kthread+0xd1/0xe0
kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
kernel: [<ffffffff91d8bd37>] ret_from_fork_nospec_begin+0x21/0x21
kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
kernel: ---[ end trace ed2119501826ec7a ]---


Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/target/iscsi/iscsi_target_util.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index 3082f5bde9fa..61233755ece0 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -741,8 +741,6 @@ void __iscsit_free_cmd(struct iscsi_cmd *cmd, bool check_queues)
 {
 	struct iscsi_conn *conn = cmd->conn;
 
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
 	if (cmd->data_direction = DMA_TO_DEVICE) {
 		iscsit_stop_dataout_timer(cmd);
 		iscsit_free_r2ts_from_list(cmd);
-- 
2.26.2

WARNING: multiple messages have this Message-ID (diff)
From: Maurizio Lombardi <mlombard@redhat.com>
To: martin.petersen@oracle.com
Cc: linux-scsi@vger.kernel.org, target-devel@vger.kernel.org,
	bvanassche@acm.org, michael.christie@oracle.com
Subject: [PATCH 2/2] target: iscsi: fix a race condition when aborting a task
Date: Wed,  7 Oct 2020 16:53:26 +0200	[thread overview]
Message-ID: <20201007145326.56850-3-mlombard@redhat.com> (raw)
In-Reply-To: <20201007145326.56850-1-mlombard@redhat.com>

The iscsit_release_commands_from_conn() function does the following
operations:

1) locks the cmd_lock spinlock
2) Scans the list of commands and sets the CMD_T_FABRIC_STOP flag
3) Releases the cmd_lock spinlock
4) Rescans the list again and clears the i_conn_node link of each command

If an abort task timer is fired between 3) and 4), it will find
the CMD_T_FABRIC_STOP flag set and won't call list_del_init();
therefore it may end up calling __iscsit_free_cmd() with a
non-empty i_conn_node list, thus triggering the warning.

Considering that:

- we expect list_del_init() to be executed by
  iscsit_release_commands_from_conn() when the CMD_T_FABRIC_STOP is set.
- iscsit_aborted_task() is the only function that calls __iscsit_free_cmd()
  directly, while all the other functions call iscsit_free_cmd().
- the warning in __iscsit_free_cmd() is a duplicate (the same warning
  can be found in iscsit_free_cmd().

We can fix the bug by simply removing the warning from __iscsit_free_cmd()

kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 1 PID: 21173 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
...
kernel: CPU: 1 PID: 21173 Comm: kworker/u8:3 Kdump: loaded Not tainted 3.10.0-1062.4.1.el7.x86_64 #1
kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
kernel: Call Trace:
kernel: [<ffffffff91d78ba4>] dump_stack+0x19/0x1b
kernel: [<ffffffff9169a958>] __warn+0xd8/0x100
kernel: [<ffffffff9169aa9d>] warn_slowpath_null+0x1d/0x20
kernel: [<ffffffffc0a6f69e>] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
kernel: [<ffffffffc0a70bc4>] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
kernel: [<ffffffffc0a7830a>] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
kernel: [<ffffffffc09fa516>] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
kernel: [<ffffffffc09f4d92>] core_tmr_abort_task+0x102/0x180 [target_core_mod]
kernel: [<ffffffffc09f7bb2>] target_tmr_work+0x152/0x170 [target_core_mod]
kernel: [<ffffffff916bd1df>] process_one_work+0x17f/0x440
kernel: [<ffffffff916be2f6>] worker_thread+0x126/0x3c0
kernel: [<ffffffff916be1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
kernel: [<ffffffff916c51b1>] kthread+0xd1/0xe0
kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
kernel: [<ffffffff91d8bd37>] ret_from_fork_nospec_begin+0x21/0x21
kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
kernel: ---[ end trace ed2119501826ec7a ]---


Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/target/iscsi/iscsi_target_util.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index 3082f5bde9fa..61233755ece0 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -741,8 +741,6 @@ void __iscsit_free_cmd(struct iscsi_cmd *cmd, bool check_queues)
 {
 	struct iscsi_conn *conn = cmd->conn;
 
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
 	if (cmd->data_direction == DMA_TO_DEVICE) {
 		iscsit_stop_dataout_timer(cmd);
 		iscsit_free_r2ts_from_list(cmd);
-- 
2.26.2


  parent reply	other threads:[~2020-10-07 14:53 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-07 14:53 [PATCH 0/2] fix race conditions with task aborts Maurizio Lombardi
2020-10-07 14:53 ` Maurizio Lombardi
2020-10-07 14:53 ` [PATCH 1/2] target: iscsi: prevent a race condition in iscsit_unmap_cmd() Maurizio Lombardi
2020-10-07 14:53   ` Maurizio Lombardi
2020-10-08  2:15   ` Bart Van Assche
2020-10-08  2:15     ` Bart Van Assche
2020-10-08  9:42     ` Maurizio Lombardi
2020-10-08  9:42       ` Maurizio Lombardi
2020-10-07 14:53 ` Maurizio Lombardi [this message]
2020-10-07 14:53   ` [PATCH 2/2] target: iscsi: fix a race condition when aborting a task Maurizio Lombardi
2020-10-22  2:42   ` Mike Christie
2020-10-22  2:42     ` Mike Christie
2020-10-27 13:49     ` Maurizio Lombardi
2020-10-27 13:49       ` Maurizio Lombardi
2020-10-27 17:54       ` Mike Christie
2020-10-27 17:54         ` Mike Christie
2020-10-27 20:03         ` Michael Christie
2020-10-27 20:03           ` Michael Christie
2020-10-28 17:09           ` Maurizio Lombardi
2020-10-28 17:09             ` Maurizio Lombardi
2020-10-28 20:37             ` Mike Christie
2020-10-28 20:37               ` Mike Christie
2020-11-10 21:29               ` Maurizio Lombardi
2020-11-10 21:29                 ` Maurizio Lombardi
2020-11-10 23:08                 ` Mike Christie
2020-11-10 23:08                   ` Mike Christie
2020-11-11  2:16             ` Mike Christie
2020-11-11  2:16               ` Mike Christie
2020-11-11 14:58               ` Maurizio Lombardi
2020-11-11 14:58                 ` Maurizio Lombardi
2020-11-11 15:37                 ` Michael Christie
2020-11-11 15:37                   ` Michael Christie
2020-11-11 15:48                   ` Maurizio Lombardi
2020-11-11 15:48                     ` Maurizio Lombardi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201007145326.56850-3-mlombard@redhat.com \
    --to=mlombard@redhat.com \
    --cc=bvanassche@acm.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=michael.christie@oracle.com \
    --cc=target-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.