[PATCH 0/3] iscsi/iser-target: Convert to kthread.h + fix logout failure

[PATCH 0/3] iscsi/iser-target: Convert to kthread.h + fix logout failure

[Nicholas A. Bellinger]

From: Nicholas Bellinger <nab@linux-iscsi.org>

Hello Sagi & Slava,

Following the earlier pre-LSF discussion, here is a patch series
to convert iscsi/iser target to using modern kthread.h primitives
when starting / stopping RX + TX kernel threads during iscsi_conn
creation / shutdown.

So far this code has been tested with traditional iscsi-target
login / logout code-paths, and needs to be verified via iser-target
login / logout code-paths as well.

Also, patch #3 is a seperate iser-target bugfix to handle a failed
outgoing ISCSI_OP_LOGOUT_RSP in isert_cq_comp_err() completion
interrupt path code.  This one may or may not have been triggering
earlier, but addresses a possible iser-target specific logout bug.

Please review + test for v4.1-rc1 code.

Thank you,

--nab

Nicholas Bellinger (3):
  iscsi-target: Convert iscsi_thread_set usage to kthread.h
  iscsi-target: Drop legacy iscsi_target_tq.c logic
  iser-target: Handle special case for logout during connection failure

 drivers/infiniband/ulp/isert/ib_isert.c   |  16 +-
 drivers/target/iscsi/Makefile             |   1 -
 drivers/target/iscsi/iscsi_target.c       | 109 +++----
 drivers/target/iscsi/iscsi_target_erl0.c  |  14 +-
 drivers/target/iscsi/iscsi_target_login.c |  60 +++-
 drivers/target/iscsi/iscsi_target_tq.c    | 495 ------------------------------
 drivers/target/iscsi/iscsi_target_tq.h    |  84 -----
 drivers/target/iscsi/iscsi_target_util.c  |   1 -
 include/target/iscsi/iscsi_target_core.h  |  13 +-
 9 files changed, 131 insertions(+), 662 deletions(-)
 delete mode 100644 drivers/target/iscsi/iscsi_target_tq.c
 delete mode 100644 drivers/target/iscsi/iscsi_target_tq.h

-- 
1.9.1

[PATCH 1/3] iscsi-target: Convert iscsi_thread_set usage to kthread.h

[Nicholas A. Bellinger]

From: Nicholas Bellinger <nab@linux-iscsi.org>

This patch converts iscsi-target code to use modern kthread.h API
callers for creating RX/TX threads for each new iscsi_conn descriptor,
and releasing associated RX/TX threads during connection shutdown.

This is done using iscsit_start_kthreads() -> kthread_run() to start
new kthreads from within iscsi_post_login_handler(), and invoking
kthread_stop() from existing iscsit_close_connection() code.

Also, convert iscsit_logout_post_handler_closesession() code to use
cmpxchg when determing when iscsit_cause_connection_reinstatement()
needs to sleep waiting for completion.

Reported-by: Sagi Grimberg <sagig@mellanox.com>
Cc: Sagi Grimberg <sagig@mellanox.com>
Cc: Slava Shwartsman <valyushash@gmail.com>
Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
---
 drivers/target/iscsi/iscsi_target.c       | 106 +++++++++++++-----------------
 drivers/target/iscsi/iscsi_target_erl0.c  |  13 ++--
 drivers/target/iscsi/iscsi_target_login.c |  59 +++++++++++++++--
 include/target/iscsi/iscsi_target_core.h  |   7 ++
 4 files changed, 116 insertions(+), 69 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 2accb6e..cb97b59 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -537,7 +537,7 @@ static struct iscsit_transport iscsi_target_transport = {
 
 static int __init iscsi_target_init_module(void)
 {
-	int ret = 0;
+	int ret = 0, size;
 
 	pr_debug("iSCSI-Target "ISCSIT_VERSION"\n");
 
@@ -546,6 +546,7 @@ static int __init iscsi_target_init_module(void)
 		pr_err("Unable to allocate memory for iscsit_global\n");
 		return -1;
 	}
+	spin_lock_init(&iscsit_global->ts_bitmap_lock);
 	mutex_init(&auth_id_lock);
 	spin_lock_init(&sess_idr_lock);
 	idr_init(&tiqn_idr);
@@ -555,15 +556,11 @@ static int __init iscsi_target_init_module(void)
 	if (ret < 0)
 		goto out;
 
-	ret = iscsi_thread_set_init();
-	if (ret < 0)
+	size = BITS_TO_LONGS(ISCSIT_BITMAP_BITS) * sizeof(long);
+	iscsit_global->ts_bitmap = vzalloc(size);
+	if (!iscsit_global->ts_bitmap) {
+		pr_err("Unable to allocate iscsit_global->ts_bitmap\n");
 		goto configfs_out;
-
-	if (iscsi_allocate_thread_sets(TARGET_THREAD_SET_COUNT) !=
-			TARGET_THREAD_SET_COUNT) {
-		pr_err("iscsi_allocate_thread_sets() returned"
-			" unexpected value!\n");
-		goto ts_out1;
 	}
 
 	lio_qr_cache = kmem_cache_create("lio_qr_cache",
@@ -572,7 +569,7 @@ static int __init iscsi_target_init_module(void)
 	if (!lio_qr_cache) {
 		pr_err("nable to kmem_cache_create() for"
 				" lio_qr_cache\n");
-		goto ts_out2;
+		goto bitmap_out;
 	}
 
 	lio_dr_cache = kmem_cache_create("lio_dr_cache",
@@ -617,10 +614,8 @@ dr_out:
 	kmem_cache_destroy(lio_dr_cache);
 qr_out:
 	kmem_cache_destroy(lio_qr_cache);
-ts_out2:
-	iscsi_deallocate_thread_sets();
-ts_out1:
-	iscsi_thread_set_free();
+bitmap_out:
+	vfree(iscsit_global->ts_bitmap);
 configfs_out:
 	iscsi_target_deregister_configfs();
 out:
@@ -630,8 +625,6 @@ out:
 
 static void __exit iscsi_target_cleanup_module(void)
 {
-	iscsi_deallocate_thread_sets();
-	iscsi_thread_set_free();
 	iscsit_release_discovery_tpg();
 	iscsit_unregister_transport(&iscsi_target_transport);
 	kmem_cache_destroy(lio_qr_cache);
@@ -641,6 +634,7 @@ static void __exit iscsi_target_cleanup_module(void)
 
 	iscsi_target_deregister_configfs();
 
+	vfree(iscsit_global->ts_bitmap);
 	kfree(iscsit_global);
 }
 
@@ -3710,17 +3704,16 @@ static int iscsit_send_reject(
 
 void iscsit_thread_get_cpumask(struct iscsi_conn *conn)
 {
-	struct iscsi_thread_set *ts = conn->thread_set;
 	int ord, cpu;
 	/*
-	 * thread_id is assigned from iscsit_global->ts_bitmap from
-	 * within iscsi_thread_set.c:iscsi_allocate_thread_sets()
+	 * bitmap_id is assigned from iscsit_global->ts_bitmap from
+	 * within iscsit_start_kthreads()
 	 *
-	 * Here we use thread_id to determine which CPU that this
-	 * iSCSI connection's iscsi_thread_set will be scheduled to
+	 * Here we use bitmap_id to determine which CPU that this
+	 * iSCSI connection's RX/TX threads will be scheduled to
 	 * execute upon.
 	 */
-	ord = ts->thread_id % cpumask_weight(cpu_online_mask);
+	ord = conn->bitmap_id % cpumask_weight(cpu_online_mask);
 	for_each_online_cpu(cpu) {
 		if (ord-- == 0) {
 			cpumask_set_cpu(cpu, conn->conn_cpumask);
@@ -3909,7 +3902,7 @@ check_rsp_state:
 	switch (state) {
 	case ISTATE_SEND_LOGOUTRSP:
 		if (!iscsit_logout_post_handler(cmd, conn))
-			goto restart;
+			return -ECONNRESET;
 		/* fall through */
 	case ISTATE_SEND_STATUS:
 	case ISTATE_SEND_ASYNCMSG:
@@ -3937,8 +3930,6 @@ check_rsp_state:
 
 err:
 	return -1;
-restart:
-	return -EAGAIN;
 }
 
 static int iscsit_handle_response_queue(struct iscsi_conn *conn)
@@ -3965,21 +3956,13 @@ static int iscsit_handle_response_queue(struct iscsi_conn *conn)
 int iscsi_target_tx_thread(void *arg)
 {
 	int ret = 0;
-	struct iscsi_conn *conn;
-	struct iscsi_thread_set *ts = arg;
+	struct iscsi_conn *conn = arg;
 	/*
 	 * Allow ourselves to be interrupted by SIGINT so that a
 	 * connection recovery / failure event can be triggered externally.
 	 */
 	allow_signal(SIGINT);
 
-restart:
-	conn = iscsi_tx_thread_pre_handler(ts);
-	if (!conn)
-		goto out;
-
-	ret = 0;
-
 	while (!kthread_should_stop()) {
 		/*
 		 * Ensure that both TX and RX per connection kthreads
@@ -3988,11 +3971,9 @@ restart:
 		iscsit_thread_check_cpumask(conn, current, 1);
 
 		wait_event_interruptible(conn->queues_wq,
-					 !iscsit_conn_all_queues_empty(conn) ||
-					 ts->status == ISCSI_THREAD_SET_RESET);
+					 !iscsit_conn_all_queues_empty(conn));
 
-		if ((ts->status == ISCSI_THREAD_SET_RESET) ||
-		     signal_pending(current))
+		if (signal_pending(current))
 			goto transport_err;
 
 get_immediate:
@@ -4003,15 +3984,14 @@ get_immediate:
 		ret = iscsit_handle_response_queue(conn);
 		if (ret == 1)
 			goto get_immediate;
-		else if (ret == -EAGAIN)
-			goto restart;
+		else if (ret == -ECONNRESET)
+			goto out;
 		else if (ret < 0)
 			goto transport_err;
 	}
 
 transport_err:
 	iscsit_take_action_for_connection_exit(conn);
-	goto restart;
 out:
 	return 0;
 }
@@ -4106,8 +4086,7 @@ int iscsi_target_rx_thread(void *arg)
 	int ret;
 	u8 buffer[ISCSI_HDR_LEN], opcode;
 	u32 checksum = 0, digest = 0;
-	struct iscsi_conn *conn = NULL;
-	struct iscsi_thread_set *ts = arg;
+	struct iscsi_conn *conn = arg;
 	struct kvec iov;
 	/*
 	 * Allow ourselves to be interrupted by SIGINT so that a
@@ -4115,11 +4094,6 @@ int iscsi_target_rx_thread(void *arg)
 	 */
 	allow_signal(SIGINT);
 
-restart:
-	conn = iscsi_rx_thread_pre_handler(ts);
-	if (!conn)
-		goto out;
-
 	if (conn->conn_transport->transport_type == ISCSI_INFINIBAND) {
 		struct completion comp;
 		int rc;
@@ -4129,7 +4103,7 @@ restart:
 		if (rc < 0)
 			goto transport_err;
 
-		goto out;
+		goto transport_err;
 	}
 
 	while (!kthread_should_stop()) {
@@ -4205,8 +4179,6 @@ transport_err:
 	if (!signal_pending(current))
 		atomic_set(&conn->transport_failed, 1);
 	iscsit_take_action_for_connection_exit(conn);
-	goto restart;
-out:
 	return 0;
 }
 
@@ -4268,7 +4240,26 @@ int iscsit_close_connection(
 	if (conn->conn_transport->transport_type == ISCSI_TCP)
 		complete(&conn->conn_logout_comp);
 
-	iscsi_release_thread_set(conn);
+	if (!strcmp(current->comm, ISCSI_RX_THREAD_NAME)) {
+		if (conn->tx_thread &&
+		    cmpxchg(&conn->tx_thread_active, true, false)) {
+			printk("close_conn signal tx_thread\n");
+			send_sig(SIGINT, conn->tx_thread, 1);
+			kthread_stop(conn->tx_thread);
+		}
+	} else if (!strcmp(current->comm, ISCSI_TX_THREAD_NAME)) {
+		if (conn->rx_thread &&
+		    cmpxchg(&conn->rx_thread_active, true, false)) {
+			printk("close_conn signal rx_thread\n");
+			send_sig(SIGINT, conn->rx_thread, 1);
+			kthread_stop(conn->rx_thread);
+		}
+	}
+
+	spin_lock(&iscsit_global->ts_bitmap_lock);
+	bitmap_release_region(iscsit_global->ts_bitmap, conn->bitmap_id,
+			      get_order(1));
+	spin_unlock(&iscsit_global->ts_bitmap_lock);
 
 	iscsit_stop_timers_for_cmds(conn);
 	iscsit_stop_nopin_response_timer(conn);
@@ -4546,15 +4537,13 @@ static void iscsit_logout_post_handler_closesession(
 	struct iscsi_conn *conn)
 {
 	struct iscsi_session *sess = conn->sess;
-
-	iscsi_set_thread_clear(conn, ISCSI_CLEAR_TX_THREAD);
-	iscsi_set_thread_set_signal(conn, ISCSI_SIGNAL_TX_THREAD);
+	int sleep = cmpxchg(&conn->tx_thread_active, true, false);
 
 	atomic_set(&conn->conn_logout_remove, 0);
 	complete(&conn->conn_logout_comp);
 
 	iscsit_dec_conn_usage_count(conn);
-	iscsit_stop_session(sess, 1, 1);
+	iscsit_stop_session(sess, sleep, sleep);
 	iscsit_dec_session_usage_count(sess);
 	target_put_session(sess->se_sess);
 }
@@ -4562,13 +4551,12 @@ static void iscsit_logout_post_handler_closesession(
 static void iscsit_logout_post_handler_samecid(
 	struct iscsi_conn *conn)
 {
-	iscsi_set_thread_clear(conn, ISCSI_CLEAR_TX_THREAD);
-	iscsi_set_thread_set_signal(conn, ISCSI_SIGNAL_TX_THREAD);
+	int sleep = cmpxchg(&conn->tx_thread_active, true, false);
 
 	atomic_set(&conn->conn_logout_remove, 0);
 	complete(&conn->conn_logout_comp);
 
-	iscsit_cause_connection_reinstatement(conn, 1);
+	iscsit_cause_connection_reinstatement(conn, sleep);
 	iscsit_dec_conn_usage_count(conn);
 }
 
diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
index bdd8731..e008ed2 100644
--- a/drivers/target/iscsi/iscsi_target_erl0.c
+++ b/drivers/target/iscsi/iscsi_target_erl0.c
@@ -860,7 +860,10 @@ void iscsit_connection_reinstatement_rcfr(struct iscsi_conn *conn)
 	}
 	spin_unlock_bh(&conn->state_lock);
 
-	iscsi_thread_set_force_reinstatement(conn);
+	if (conn->tx_thread && conn->tx_thread_active)
+		send_sig(SIGINT, conn->tx_thread, 1);
+	if (conn->rx_thread && conn->rx_thread_active)
+		send_sig(SIGINT, conn->rx_thread, 1);
 
 sleep:
 	wait_for_completion(&conn->conn_wait_rcfr_comp);
@@ -885,10 +888,10 @@ void iscsit_cause_connection_reinstatement(struct iscsi_conn *conn, int sleep)
 		return;
 	}
 
-	if (iscsi_thread_set_force_reinstatement(conn) < 0) {
-		spin_unlock_bh(&conn->state_lock);
-		return;
-	}
+	if (conn->tx_thread && conn->tx_thread_active)
+		send_sig(SIGINT, conn->tx_thread, 1);
+	if (conn->rx_thread && conn->rx_thread_active)
+		send_sig(SIGINT, conn->rx_thread, 1);
 
 	atomic_set(&conn->connection_reinstatement, 1);
 	if (!sleep) {
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index 153fb66..345f073 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -699,6 +699,51 @@ static void iscsi_post_login_start_timers(struct iscsi_conn *conn)
 		iscsit_start_nopin_timer(conn);
 }
 
+int iscsit_start_kthreads(struct iscsi_conn *conn)
+{
+	int ret = 0;
+
+	spin_lock(&iscsit_global->ts_bitmap_lock);
+	conn->bitmap_id = bitmap_find_free_region(iscsit_global->ts_bitmap,
+					ISCSIT_BITMAP_BITS, get_order(1));
+	spin_unlock(&iscsit_global->ts_bitmap_lock);
+
+	if (conn->bitmap_id < 0) {
+		pr_err("bitmap_find_free_region() failed for"
+		       " iscsit_start_kthreads()\n");
+		return -ENOMEM;
+	}
+
+	conn->tx_thread = kthread_run(iscsi_target_tx_thread, conn,
+				      "%s", ISCSI_TX_THREAD_NAME);
+	if (IS_ERR(conn->tx_thread)) {
+		pr_err("Unable to start iscsi_target_tx_thread\n");
+		ret = PTR_ERR(conn->tx_thread);
+		goto out_bitmap;
+	}
+	conn->tx_thread_active = true;
+
+	conn->rx_thread = kthread_run(iscsi_target_rx_thread, conn,
+				      "%s", ISCSI_RX_THREAD_NAME);
+	if (IS_ERR(conn->rx_thread)) {
+		pr_err("Unable to start iscsi_target_rx_thread\n");
+		ret = PTR_ERR(conn->rx_thread);
+		goto out_tx;
+	}
+	conn->rx_thread_active = true;
+
+	return 0;
+out_tx:
+	kthread_stop(conn->tx_thread);
+	conn->tx_thread_active = false;
+out_bitmap:
+	spin_lock(&iscsit_global->ts_bitmap_lock);
+	bitmap_release_region(iscsit_global->ts_bitmap, conn->bitmap_id,
+			      get_order(1));
+	spin_unlock(&iscsit_global->ts_bitmap_lock);
+	return ret;
+}
+
 int iscsi_post_login_handler(
 	struct iscsi_np *np,
 	struct iscsi_conn *conn,
@@ -709,7 +754,7 @@ int iscsi_post_login_handler(
 	struct se_session *se_sess = sess->se_sess;
 	struct iscsi_portal_group *tpg = sess->tpg;
 	struct se_portal_group *se_tpg = &tpg->tpg_se_tpg;
-	struct iscsi_thread_set *ts;
+	int rc;
 
 	iscsit_inc_conn_usage_count(conn);
 
@@ -724,7 +769,6 @@ int iscsi_post_login_handler(
 	/*
 	 * SCSI Initiator -> SCSI Target Port Mapping
 	 */
-	ts = iscsi_get_thread_set();
 	if (!zero_tsih) {
 		iscsi_set_session_parameters(sess->sess_ops,
 				conn->param_list, 0);
@@ -751,9 +795,11 @@ int iscsi_post_login_handler(
 			sess->sess_ops->InitiatorName);
 		spin_unlock_bh(&sess->conn_lock);
 
-		iscsi_post_login_start_timers(conn);
+		rc = iscsit_start_kthreads(conn);
+		if (rc)
+			return rc;
 
-		iscsi_activate_thread_set(conn, ts);
+		iscsi_post_login_start_timers(conn);
 		/*
 		 * Determine CPU mask to ensure connection's RX and TX kthreads
 		 * are scheduled on the same CPU.
@@ -810,8 +856,11 @@ int iscsi_post_login_handler(
 		" iSCSI Target Portal Group: %hu\n", tpg->nsessions, tpg->tpgt);
 	spin_unlock_bh(&se_tpg->session_lock);
 
+	rc = iscsit_start_kthreads(conn);
+	if (rc)
+		return rc;
+
 	iscsi_post_login_start_timers(conn);
-	iscsi_activate_thread_set(conn, ts);
 	/*
 	 * Determine CPU mask to ensure connection's RX and TX kthreads
 	 * are scheduled on the same CPU.
diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h
index d3583d3..dd0f3ab 100644
--- a/include/target/iscsi/iscsi_target_core.h
+++ b/include/target/iscsi/iscsi_target_core.h
@@ -602,6 +602,11 @@ struct iscsi_conn {
 	struct iscsi_session	*sess;
 	/* Pointer to thread_set in use for this conn's threads */
 	struct iscsi_thread_set	*thread_set;
+	int			bitmap_id;
+	int			rx_thread_active;
+	struct task_struct	*rx_thread;
+	int			tx_thread_active;
+	struct task_struct	*tx_thread;
 	/* list_head for session connection list */
 	struct list_head	conn_list;
 } ____cacheline_aligned;
@@ -871,10 +876,12 @@ struct iscsit_global {
 	/* Unique identifier used for the authentication daemon */
 	u32			auth_id;
 	u32			inactive_ts;
+#define ISCSIT_BITMAP_BITS	262144
 	/* Thread Set bitmap count */
 	int			ts_bitmap_count;
 	/* Thread Set bitmap pointer */
 	unsigned long		*ts_bitmap;
+	spinlock_t		ts_bitmap_lock;
 	/* Used for iSCSI discovery session authentication */
 	struct iscsi_node_acl	discovery_acl;
 	struct iscsi_portal_group	*discovery_tpg;
-- 
1.9.1

[PATCH 2/3] iscsi-target: Drop legacy iscsi_target_tq.c logic

[Nicholas A. Bellinger]

From: Nicholas Bellinger <nab@linux-iscsi.org>

Now that iscsi_conn allocates new [rx,tx] threads using kthread.h
primitives on the fly, and kthread_stop() is called directly during
connection shutdown, it's time to go ahead and drop iscsi_target_tq.c
legacy code.

The use of multiple struct completion in iscsi_activate_thread_set()
has been proven to cause issues during repeated iser login/logout.

Cc: Sagi Grimberg <sagig@mellanox.com>
Cc: Slava Shwartsman <valyushash@gmail.com>
Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
---
 drivers/target/iscsi/Makefile             |   1 -
 drivers/target/iscsi/iscsi_target.c       |   3 -
 drivers/target/iscsi/iscsi_target_erl0.c  |   1 -
 drivers/target/iscsi/iscsi_target_login.c |   1 -
 drivers/target/iscsi/iscsi_target_tq.c    | 495 ------------------------------
 drivers/target/iscsi/iscsi_target_tq.h    |  84 -----
 drivers/target/iscsi/iscsi_target_util.c  |   1 -
 include/target/iscsi/iscsi_target_core.h  |   6 +-
 8 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 drivers/target/iscsi/iscsi_target_tq.c
 delete mode 100644 drivers/target/iscsi/iscsi_target_tq.h

diff --git a/drivers/target/iscsi/Makefile b/drivers/target/iscsi/Makefile
index 13a9240..0f43be9 100644
--- a/drivers/target/iscsi/Makefile
+++ b/drivers/target/iscsi/Makefile
@@ -1,6 +1,5 @@
 iscsi_target_mod-y +=		iscsi_target_parameters.o \
 				iscsi_target_seq_pdu_list.o \
-				iscsi_target_tq.o \
 				iscsi_target_auth.o \
 				iscsi_target_datain_values.o \
 				iscsi_target_device.o \
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index cb97b59..ca731b5 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -33,7 +33,6 @@
 #include <target/iscsi/iscsi_target_core.h>
 #include "iscsi_target_parameters.h"
 #include "iscsi_target_seq_pdu_list.h"
-#include "iscsi_target_tq.h"
 #include "iscsi_target_configfs.h"
 #include "iscsi_target_datain_values.h"
 #include "iscsi_target_erl0.h"
@@ -4369,8 +4368,6 @@ int iscsit_close_connection(
 
 	iscsit_put_transport(conn->conn_transport);
 
-	conn->thread_set = NULL;
-
 	pr_debug("Moving to TARG_CONN_STATE_FREE.\n");
 	conn->conn_state = TARG_CONN_STATE_FREE;
 	kfree(conn);
diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
index e008ed2..959a14c 100644
--- a/drivers/target/iscsi/iscsi_target_erl0.c
+++ b/drivers/target/iscsi/iscsi_target_erl0.c
@@ -23,7 +23,6 @@
 
 #include <target/iscsi/iscsi_target_core.h>
 #include "iscsi_target_seq_pdu_list.h"
-#include "iscsi_target_tq.h"
 #include "iscsi_target_erl0.h"
 #include "iscsi_target_erl1.h"
 #include "iscsi_target_erl2.h"
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index 345f073..af20ddf 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -26,7 +26,6 @@
 
 #include <target/iscsi/iscsi_target_core.h>
 #include <target/iscsi/iscsi_target_stat.h>
-#include "iscsi_target_tq.h"
 #include "iscsi_target_device.h"
 #include "iscsi_target_nego.h"
 #include "iscsi_target_erl0.h"
diff --git a/drivers/target/iscsi/iscsi_target_tq.c b/drivers/target/iscsi/iscsi_target_tq.c
deleted file mode 100644
index 26aa509..0000000
--- a/drivers/target/iscsi/iscsi_target_tq.c
+++ /dev/null
@@ -1,495 +0,0 @@
-/*******************************************************************************
- * This file contains the iSCSI Login Thread and Thread Queue functions.
- *
- * (c) Copyright 2007-2013 Datera, Inc.
- *
- * Author: Nicholas A. Bellinger <nab@linux-iscsi.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- ******************************************************************************/
-
-#include <linux/kthread.h>
-#include <linux/list.h>
-#include <linux/bitmap.h>
-
-#include <target/iscsi/iscsi_target_core.h>
-#include "iscsi_target_tq.h"
-#include "iscsi_target.h"
-
-static LIST_HEAD(inactive_ts_list);
-static DEFINE_SPINLOCK(inactive_ts_lock);
-static DEFINE_SPINLOCK(ts_bitmap_lock);
-
-static void iscsi_add_ts_to_inactive_list(struct iscsi_thread_set *ts)
-{
-	if (!list_empty(&ts->ts_list)) {
-		WARN_ON(1);
-		return;
-	}
-	spin_lock(&inactive_ts_lock);
-	list_add_tail(&ts->ts_list, &inactive_ts_list);
-	iscsit_global->inactive_ts++;
-	spin_unlock(&inactive_ts_lock);
-}
-
-static struct iscsi_thread_set *iscsi_get_ts_from_inactive_list(void)
-{
-	struct iscsi_thread_set *ts;
-
-	spin_lock(&inactive_ts_lock);
-	if (list_empty(&inactive_ts_list)) {
-		spin_unlock(&inactive_ts_lock);
-		return NULL;
-	}
-
-	ts = list_first_entry(&inactive_ts_list, struct iscsi_thread_set, ts_list);
-
-	list_del_init(&ts->ts_list);
-	iscsit_global->inactive_ts--;
-	spin_unlock(&inactive_ts_lock);
-
-	return ts;
-}
-
-int iscsi_allocate_thread_sets(u32 thread_pair_count)
-{
-	int allocated_thread_pair_count = 0, i, thread_id;
-	struct iscsi_thread_set *ts = NULL;
-
-	for (i = 0; i < thread_pair_count; i++) {
-		ts = kzalloc(sizeof(struct iscsi_thread_set), GFP_KERNEL);
-		if (!ts) {
-			pr_err("Unable to allocate memory for"
-					" thread set.\n");
-			return allocated_thread_pair_count;
-		}
-		/*
-		 * Locate the next available regision in the thread_set_bitmap
-		 */
-		spin_lock(&ts_bitmap_lock);
-		thread_id = bitmap_find_free_region(iscsit_global->ts_bitmap,
-				iscsit_global->ts_bitmap_count, get_order(1));
-		spin_unlock(&ts_bitmap_lock);
-		if (thread_id < 0) {
-			pr_err("bitmap_find_free_region() failed for"
-				" thread_set_bitmap\n");
-			kfree(ts);
-			return allocated_thread_pair_count;
-		}
-
-		ts->thread_id = thread_id;
-		ts->status = ISCSI_THREAD_SET_FREE;
-		INIT_LIST_HEAD(&ts->ts_list);
-		spin_lock_init(&ts->ts_state_lock);
-		init_completion(&ts->rx_restart_comp);
-		init_completion(&ts->tx_restart_comp);
-		init_completion(&ts->rx_start_comp);
-		init_completion(&ts->tx_start_comp);
-		sema_init(&ts->ts_activate_sem, 0);
-
-		ts->create_threads = 1;
-		ts->tx_thread = kthread_run(iscsi_target_tx_thread, ts, "%s",
-					ISCSI_TX_THREAD_NAME);
-		if (IS_ERR(ts->tx_thread)) {
-			dump_stack();
-			pr_err("Unable to start iscsi_target_tx_thread\n");
-			break;
-		}
-
-		ts->rx_thread = kthread_run(iscsi_target_rx_thread, ts, "%s",
-					ISCSI_RX_THREAD_NAME);
-		if (IS_ERR(ts->rx_thread)) {
-			kthread_stop(ts->tx_thread);
-			pr_err("Unable to start iscsi_target_rx_thread\n");
-			break;
-		}
-		ts->create_threads = 0;
-
-		iscsi_add_ts_to_inactive_list(ts);
-		allocated_thread_pair_count++;
-	}
-
-	pr_debug("Spawned %d thread set(s) (%d total threads).\n",
-		allocated_thread_pair_count, allocated_thread_pair_count * 2);
-	return allocated_thread_pair_count;
-}
-
-static void iscsi_deallocate_thread_one(struct iscsi_thread_set *ts)
-{
-	spin_lock_bh(&ts->ts_state_lock);
-	ts->status = ISCSI_THREAD_SET_DIE;
-
-	if (ts->rx_thread) {
-		complete(&ts->rx_start_comp);
-		spin_unlock_bh(&ts->ts_state_lock);
-		kthread_stop(ts->rx_thread);
-		spin_lock_bh(&ts->ts_state_lock);
-	}
-	if (ts->tx_thread) {
-		complete(&ts->tx_start_comp);
-		spin_unlock_bh(&ts->ts_state_lock);
-		kthread_stop(ts->tx_thread);
-		spin_lock_bh(&ts->ts_state_lock);
-	}
-	spin_unlock_bh(&ts->ts_state_lock);
-	/*
-	 * Release this thread_id in the thread_set_bitmap
-	 */
-	spin_lock(&ts_bitmap_lock);
-	bitmap_release_region(iscsit_global->ts_bitmap,
-			ts->thread_id, get_order(1));
-	spin_unlock(&ts_bitmap_lock);
-
-	kfree(ts);
-}
-
-void iscsi_deallocate_thread_sets(void)
-{
-	struct iscsi_thread_set *ts = NULL;
-	u32 released_count = 0;
-
-	while ((ts = iscsi_get_ts_from_inactive_list())) {
-
-		iscsi_deallocate_thread_one(ts);
-		released_count++;
-	}
-
-	if (released_count)
-		pr_debug("Stopped %d thread set(s) (%d total threads)."
-			"\n", released_count, released_count * 2);
-}
-
-static void iscsi_deallocate_extra_thread_sets(void)
-{
-	u32 orig_count, released_count = 0;
-	struct iscsi_thread_set *ts = NULL;
-
-	orig_count = TARGET_THREAD_SET_COUNT;
-
-	while ((iscsit_global->inactive_ts + 1) > orig_count) {
-		ts = iscsi_get_ts_from_inactive_list();
-		if (!ts)
-			break;
-
-		iscsi_deallocate_thread_one(ts);
-		released_count++;
-	}
-
-	if (released_count)
-		pr_debug("Stopped %d thread set(s) (%d total threads)."
-			"\n", released_count, released_count * 2);
-}
-
-void iscsi_activate_thread_set(struct iscsi_conn *conn, struct iscsi_thread_set *ts)
-{
-	spin_lock_bh(&ts->ts_state_lock);
-	conn->thread_set = ts;
-	ts->conn = conn;
-	ts->status = ISCSI_THREAD_SET_ACTIVE;
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	complete(&ts->rx_start_comp);
-	complete(&ts->tx_start_comp);
-
-	down(&ts->ts_activate_sem);
-}
-
-struct iscsi_thread_set *iscsi_get_thread_set(void)
-{
-	struct iscsi_thread_set *ts;
-
-get_set:
-	ts = iscsi_get_ts_from_inactive_list();
-	if (!ts) {
-		iscsi_allocate_thread_sets(1);
-		goto get_set;
-	}
-
-	ts->delay_inactive = 1;
-	ts->signal_sent = 0;
-	ts->thread_count = 2;
-	init_completion(&ts->rx_restart_comp);
-	init_completion(&ts->tx_restart_comp);
-	sema_init(&ts->ts_activate_sem, 0);
-
-	return ts;
-}
-
-void iscsi_set_thread_clear(struct iscsi_conn *conn, u8 thread_clear)
-{
-	struct iscsi_thread_set *ts = NULL;
-
-	if (!conn->thread_set) {
-		pr_err("struct iscsi_conn->thread_set is NULL\n");
-		return;
-	}
-	ts = conn->thread_set;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	ts->thread_clear &= ~thread_clear;
-
-	if ((thread_clear & ISCSI_CLEAR_RX_THREAD) &&
-	    (ts->blocked_threads & ISCSI_BLOCK_RX_THREAD))
-		complete(&ts->rx_restart_comp);
-	else if ((thread_clear & ISCSI_CLEAR_TX_THREAD) &&
-		 (ts->blocked_threads & ISCSI_BLOCK_TX_THREAD))
-		complete(&ts->tx_restart_comp);
-	spin_unlock_bh(&ts->ts_state_lock);
-}
-
-void iscsi_set_thread_set_signal(struct iscsi_conn *conn, u8 signal_sent)
-{
-	struct iscsi_thread_set *ts = NULL;
-
-	if (!conn->thread_set) {
-		pr_err("struct iscsi_conn->thread_set is NULL\n");
-		return;
-	}
-	ts = conn->thread_set;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	ts->signal_sent |= signal_sent;
-	spin_unlock_bh(&ts->ts_state_lock);
-}
-
-int iscsi_release_thread_set(struct iscsi_conn *conn)
-{
-	int thread_called = 0;
-	struct iscsi_thread_set *ts = NULL;
-
-	if (!conn || !conn->thread_set) {
-		pr_err("connection or thread set pointer is NULL\n");
-		BUG();
-	}
-	ts = conn->thread_set;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	ts->status = ISCSI_THREAD_SET_RESET;
-
-	if (!strncmp(current->comm, ISCSI_RX_THREAD_NAME,
-			strlen(ISCSI_RX_THREAD_NAME)))
-		thread_called = ISCSI_RX_THREAD;
-	else if (!strncmp(current->comm, ISCSI_TX_THREAD_NAME,
-			strlen(ISCSI_TX_THREAD_NAME)))
-		thread_called = ISCSI_TX_THREAD;
-
-	if (ts->rx_thread && (thread_called == ISCSI_TX_THREAD) &&
-	   (ts->thread_clear & ISCSI_CLEAR_RX_THREAD)) {
-
-		if (!(ts->signal_sent & ISCSI_SIGNAL_RX_THREAD)) {
-			send_sig(SIGINT, ts->rx_thread, 1);
-			ts->signal_sent |= ISCSI_SIGNAL_RX_THREAD;
-		}
-		ts->blocked_threads |= ISCSI_BLOCK_RX_THREAD;
-		spin_unlock_bh(&ts->ts_state_lock);
-		wait_for_completion(&ts->rx_restart_comp);
-		spin_lock_bh(&ts->ts_state_lock);
-		ts->blocked_threads &= ~ISCSI_BLOCK_RX_THREAD;
-	}
-	if (ts->tx_thread && (thread_called == ISCSI_RX_THREAD) &&
-	   (ts->thread_clear & ISCSI_CLEAR_TX_THREAD)) {
-
-		if (!(ts->signal_sent & ISCSI_SIGNAL_TX_THREAD)) {
-			send_sig(SIGINT, ts->tx_thread, 1);
-			ts->signal_sent |= ISCSI_SIGNAL_TX_THREAD;
-		}
-		ts->blocked_threads |= ISCSI_BLOCK_TX_THREAD;
-		spin_unlock_bh(&ts->ts_state_lock);
-		wait_for_completion(&ts->tx_restart_comp);
-		spin_lock_bh(&ts->ts_state_lock);
-		ts->blocked_threads &= ~ISCSI_BLOCK_TX_THREAD;
-	}
-
-	ts->conn = NULL;
-	ts->status = ISCSI_THREAD_SET_FREE;
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	return 0;
-}
-
-int iscsi_thread_set_force_reinstatement(struct iscsi_conn *conn)
-{
-	struct iscsi_thread_set *ts;
-
-	if (!conn->thread_set)
-		return -1;
-	ts = conn->thread_set;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	if (ts->status != ISCSI_THREAD_SET_ACTIVE) {
-		spin_unlock_bh(&ts->ts_state_lock);
-		return -1;
-	}
-
-	if (ts->tx_thread && (!(ts->signal_sent & ISCSI_SIGNAL_TX_THREAD))) {
-		send_sig(SIGINT, ts->tx_thread, 1);
-		ts->signal_sent |= ISCSI_SIGNAL_TX_THREAD;
-	}
-	if (ts->rx_thread && (!(ts->signal_sent & ISCSI_SIGNAL_RX_THREAD))) {
-		send_sig(SIGINT, ts->rx_thread, 1);
-		ts->signal_sent |= ISCSI_SIGNAL_RX_THREAD;
-	}
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	return 0;
-}
-
-static void iscsi_check_to_add_additional_sets(void)
-{
-	int thread_sets_add;
-
-	spin_lock(&inactive_ts_lock);
-	thread_sets_add = iscsit_global->inactive_ts;
-	spin_unlock(&inactive_ts_lock);
-	if (thread_sets_add == 1)
-		iscsi_allocate_thread_sets(1);
-}
-
-static int iscsi_signal_thread_pre_handler(struct iscsi_thread_set *ts)
-{
-	spin_lock_bh(&ts->ts_state_lock);
-	if (ts->status == ISCSI_THREAD_SET_DIE || kthread_should_stop() ||
-	    signal_pending(current)) {
-		spin_unlock_bh(&ts->ts_state_lock);
-		return -1;
-	}
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	return 0;
-}
-
-struct iscsi_conn *iscsi_rx_thread_pre_handler(struct iscsi_thread_set *ts)
-{
-	int ret;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	if (ts->create_threads) {
-		spin_unlock_bh(&ts->ts_state_lock);
-		goto sleep;
-	}
-
-	if (ts->status != ISCSI_THREAD_SET_DIE)
-		flush_signals(current);
-
-	if (ts->delay_inactive && (--ts->thread_count == 0)) {
-		spin_unlock_bh(&ts->ts_state_lock);
-
-		if (!iscsit_global->in_shutdown)
-			iscsi_deallocate_extra_thread_sets();
-
-		iscsi_add_ts_to_inactive_list(ts);
-		spin_lock_bh(&ts->ts_state_lock);
-	}
-
-	if ((ts->status == ISCSI_THREAD_SET_RESET) &&
-	    (ts->thread_clear & ISCSI_CLEAR_RX_THREAD))
-		complete(&ts->rx_restart_comp);
-
-	ts->thread_clear &= ~ISCSI_CLEAR_RX_THREAD;
-	spin_unlock_bh(&ts->ts_state_lock);
-sleep:
-	ret = wait_for_completion_interruptible(&ts->rx_start_comp);
-	if (ret != 0)
-		return NULL;
-
-	if (iscsi_signal_thread_pre_handler(ts) < 0)
-		return NULL;
-
-	iscsi_check_to_add_additional_sets();
-
-	spin_lock_bh(&ts->ts_state_lock);
-	if (!ts->conn) {
-		pr_err("struct iscsi_thread_set->conn is NULL for"
-			" RX thread_id: %s/%d\n", current->comm, current->pid);
-		spin_unlock_bh(&ts->ts_state_lock);
-		return NULL;
-	}
-	ts->thread_clear |= ISCSI_CLEAR_RX_THREAD;
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	up(&ts->ts_activate_sem);
-
-	return ts->conn;
-}
-
-struct iscsi_conn *iscsi_tx_thread_pre_handler(struct iscsi_thread_set *ts)
-{
-	int ret;
-
-	spin_lock_bh(&ts->ts_state_lock);
-	if (ts->create_threads) {
-		spin_unlock_bh(&ts->ts_state_lock);
-		goto sleep;
-	}
-
-	if (ts->status != ISCSI_THREAD_SET_DIE)
-		flush_signals(current);
-
-	if (ts->delay_inactive && (--ts->thread_count == 0)) {
-		spin_unlock_bh(&ts->ts_state_lock);
-
-		if (!iscsit_global->in_shutdown)
-			iscsi_deallocate_extra_thread_sets();
-
-		iscsi_add_ts_to_inactive_list(ts);
-		spin_lock_bh(&ts->ts_state_lock);
-	}
-	if ((ts->status == ISCSI_THREAD_SET_RESET) &&
-	    (ts->thread_clear & ISCSI_CLEAR_TX_THREAD))
-		complete(&ts->tx_restart_comp);
-
-	ts->thread_clear &= ~ISCSI_CLEAR_TX_THREAD;
-	spin_unlock_bh(&ts->ts_state_lock);
-sleep:
-	ret = wait_for_completion_interruptible(&ts->tx_start_comp);
-	if (ret != 0)
-		return NULL;
-
-	if (iscsi_signal_thread_pre_handler(ts) < 0)
-		return NULL;
-
-	iscsi_check_to_add_additional_sets();
-
-	spin_lock_bh(&ts->ts_state_lock);
-	if (!ts->conn) {
-		pr_err("struct iscsi_thread_set->conn is NULL for"
-			" TX thread_id: %s/%d\n", current->comm, current->pid);
-		spin_unlock_bh(&ts->ts_state_lock);
-		return NULL;
-	}
-	ts->thread_clear |= ISCSI_CLEAR_TX_THREAD;
-	spin_unlock_bh(&ts->ts_state_lock);
-
-	up(&ts->ts_activate_sem);
-
-	return ts->conn;
-}
-
-int iscsi_thread_set_init(void)
-{
-	int size;
-
-	iscsit_global->ts_bitmap_count = ISCSI_TS_BITMAP_BITS;
-
-	size = BITS_TO_LONGS(iscsit_global->ts_bitmap_count) * sizeof(long);
-	iscsit_global->ts_bitmap = kzalloc(size, GFP_KERNEL);
-	if (!iscsit_global->ts_bitmap) {
-		pr_err("Unable to allocate iscsit_global->ts_bitmap\n");
-		return -ENOMEM;
-	}
-
-	return 0;
-}
-
-void iscsi_thread_set_free(void)
-{
-	kfree(iscsit_global->ts_bitmap);
-}
diff --git a/drivers/target/iscsi/iscsi_target_tq.h b/drivers/target/iscsi/iscsi_target_tq.h
deleted file mode 100644
index cc1eede..0000000
--- a/drivers/target/iscsi/iscsi_target_tq.h
+++ /dev/null
@@ -1,84 +0,0 @@
-#ifndef ISCSI_THREAD_QUEUE_H
-#define ISCSI_THREAD_QUEUE_H
-
-/*
- * Defines for thread sets.
- */
-extern int iscsi_thread_set_force_reinstatement(struct iscsi_conn *);
-extern int iscsi_allocate_thread_sets(u32);
-extern void iscsi_deallocate_thread_sets(void);
-extern void iscsi_activate_thread_set(struct iscsi_conn *, struct iscsi_thread_set *);
-extern struct iscsi_thread_set *iscsi_get_thread_set(void);
-extern void iscsi_set_thread_clear(struct iscsi_conn *, u8);
-extern void iscsi_set_thread_set_signal(struct iscsi_conn *, u8);
-extern int iscsi_release_thread_set(struct iscsi_conn *);
-extern struct iscsi_conn *iscsi_rx_thread_pre_handler(struct iscsi_thread_set *);
-extern struct iscsi_conn *iscsi_tx_thread_pre_handler(struct iscsi_thread_set *);
-extern int iscsi_thread_set_init(void);
-extern void iscsi_thread_set_free(void);
-
-extern int iscsi_target_tx_thread(void *);
-extern int iscsi_target_rx_thread(void *);
-
-#define TARGET_THREAD_SET_COUNT			4
-
-#define ISCSI_RX_THREAD                         1
-#define ISCSI_TX_THREAD                         2
-#define ISCSI_RX_THREAD_NAME			"iscsi_trx"
-#define ISCSI_TX_THREAD_NAME			"iscsi_ttx"
-#define ISCSI_BLOCK_RX_THREAD			0x1
-#define ISCSI_BLOCK_TX_THREAD			0x2
-#define ISCSI_CLEAR_RX_THREAD			0x1
-#define ISCSI_CLEAR_TX_THREAD			0x2
-#define ISCSI_SIGNAL_RX_THREAD			0x1
-#define ISCSI_SIGNAL_TX_THREAD			0x2
-
-/* struct iscsi_thread_set->status */
-#define ISCSI_THREAD_SET_FREE			1
-#define ISCSI_THREAD_SET_ACTIVE			2
-#define ISCSI_THREAD_SET_DIE			3
-#define ISCSI_THREAD_SET_RESET			4
-#define ISCSI_THREAD_SET_DEALLOCATE_THREADS	5
-
-/* By default allow a maximum of 32K iSCSI connections */
-#define ISCSI_TS_BITMAP_BITS			32768
-
-struct iscsi_thread_set {
-	/* flags used for blocking and restarting sets */
-	int	blocked_threads;
-	/* flag for creating threads */
-	int	create_threads;
-	/* flag for delaying readding to inactive list */
-	int	delay_inactive;
-	/* status for thread set */
-	int	status;
-	/* which threads have had signals sent */
-	int	signal_sent;
-	/* flag for which threads exited first */
-	int	thread_clear;
-	/* Active threads in the thread set */
-	int	thread_count;
-	/* Unique thread ID */
-	u32	thread_id;
-	/* pointer to connection if set is active */
-	struct iscsi_conn	*conn;
-	/* used for controlling ts state accesses */
-	spinlock_t	ts_state_lock;
-	/* used for restarting thread queue */
-	struct completion	rx_restart_comp;
-	/* used for restarting thread queue */
-	struct completion	tx_restart_comp;
-	/* used for normal unused blocking */
-	struct completion	rx_start_comp;
-	/* used for normal unused blocking */
-	struct completion	tx_start_comp;
-	/* OS descriptor for rx thread */
-	struct task_struct	*rx_thread;
-	/* OS descriptor for tx thread */
-	struct task_struct	*tx_thread;
-	/* struct iscsi_thread_set in list list head*/
-	struct list_head	ts_list;
-	struct semaphore	ts_activate_sem;
-};
-
-#endif   /*** ISCSI_THREAD_QUEUE_H ***/
diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index 390df8e..b18edda 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -33,7 +33,6 @@
 #include "iscsi_target_erl1.h"
 #include "iscsi_target_erl2.h"
 #include "iscsi_target_tpg.h"
-#include "iscsi_target_tq.h"
 #include "iscsi_target_util.h"
 #include "iscsi_target.h"
 
diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h
index dd0f3ab..0e394a0 100644
--- a/include/target/iscsi/iscsi_target_core.h
+++ b/include/target/iscsi/iscsi_target_core.h
@@ -20,6 +20,8 @@
 #define ISCSIT_MIN_TAGS			16
 #define ISCSIT_EXTRA_TAGS		8
 #define ISCSIT_TCP_BACKLOG		256
+#define ISCSI_RX_THREAD_NAME		"iscsi_trx"
+#define ISCSI_TX_THREAD_NAME		"iscsi_ttx"
 
 /* struct iscsi_node_attrib sanity values */
 #define NA_DATAOUT_TIMEOUT		3
@@ -600,8 +602,6 @@ struct iscsi_conn {
 	struct iscsi_tpg_np	*tpg_np;
 	/* Pointer to parent session */
 	struct iscsi_session	*sess;
-	/* Pointer to thread_set in use for this conn's threads */
-	struct iscsi_thread_set	*thread_set;
 	int			bitmap_id;
 	int			rx_thread_active;
 	struct task_struct	*rx_thread;
@@ -877,8 +877,6 @@ struct iscsit_global {
 	u32			auth_id;
 	u32			inactive_ts;
 #define ISCSIT_BITMAP_BITS	262144
-	/* Thread Set bitmap count */
-	int			ts_bitmap_count;
 	/* Thread Set bitmap pointer */
 	unsigned long		*ts_bitmap;
 	spinlock_t		ts_bitmap_lock;
-- 
1.9.1

[PATCH 3/3] iser-target: Handle special case for logout during connection failure

[Nicholas A. Bellinger]

From: Nicholas Bellinger <nab@linux-iscsi.org>

This patch adds special case handling during ISCSI_OP_LOGIN_RSP ib_wr
failure, where isert_cq_comp_err() is responsible for calling the
remaining isert_response_completion() -> isert_do_control_comp() ->
iscsit_logout_post_handler() to drop the last iscsi_conn reference.

It fixes a bug where iscsit_logout_post_handler() would not be called
if the outgoing ISCSI_OP_LOGIN_RSP failed during iscsi_conn ib_wr
descriptor cleanup.

Reported-by: Sagi Grimberg <sagig@mellanox.com>
Reported-by: Slava Shwartsman <valyushash@gmail.com>
Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 075b19c..ff0d436 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1996,10 +1996,20 @@ isert_cq_comp_err(struct isert_conn *isert_conn, struct ib_wc *wc)
 
 		desc = (struct iser_tx_desc *)(uintptr_t)wc->wr_id;
 		isert_cmd = desc->isert_cmd;
-		if (!isert_cmd)
+		if (!isert_cmd) {
 			isert_unmap_tx_desc(desc, ib_dev);
-		else
-			isert_completion_put(desc, isert_cmd, ib_dev, true);
+		} else {
+			struct isert_device *device = isert_conn->conn_device;
+			struct iscsi_conn *conn = isert_conn->conn;
+			struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+
+			if (cmd->i_state == ISTATE_SEND_LOGOUTRSP &&
+			    conn->conn_state == TARG_CONN_STATE_IN_LOGOUT)
+				isert_response_completion(desc, isert_cmd, isert_conn,
+							  device->ib_device);
+			else
+				isert_completion_put(desc, isert_cmd, ib_dev, true);
+		}
 	} else {
 		isert_conn->post_recv_buf_count--;
 		if (!isert_conn->post_recv_buf_count)
-- 
1.9.1

Re: [PATCH 3/3] iser-target: Handle special case for logout during connection failure

[Sagi Grimberg]

On 3/21/2015 8:16 AM, Nicholas A. Bellinger wrote:
> From: Nicholas Bellinger <nab@linux-iscsi.org>
>
> This patch adds special case handling during ISCSI_OP_LOGIN_RSP ib_wr
> failure, where isert_cq_comp_err() is responsible for calling the
> remaining isert_response_completion() -> isert_do_control_comp() ->
> iscsit_logout_post_handler() to drop the last iscsi_conn reference.
>
> It fixes a bug where iscsit_logout_post_handler() would not be called
> if the outgoing ISCSI_OP_LOGIN_RSP failed during iscsi_conn ib_wr
> descriptor cleanup.
>
> Reported-by: Sagi Grimberg <sagig@mellanox.com>
> Reported-by: Slava Shwartsman <valyushash@gmail.com>
> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
> ---
>   drivers/infiniband/ulp/isert/ib_isert.c | 16 +++++++++++++---
>   1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
> index 075b19c..ff0d436 100644
> --- a/drivers/infiniband/ulp/isert/ib_isert.c
> +++ b/drivers/infiniband/ulp/isert/ib_isert.c
> @@ -1996,10 +1996,20 @@ isert_cq_comp_err(struct isert_conn *isert_conn, struct ib_wc *wc)
>
>   		desc = (struct iser_tx_desc *)(uintptr_t)wc->wr_id;
>   		isert_cmd = desc->isert_cmd;
> -		if (!isert_cmd)
> +		if (!isert_cmd) {
>   			isert_unmap_tx_desc(desc, ib_dev);
> -		else
> -			isert_completion_put(desc, isert_cmd, ib_dev, true);
> +		} else {
> +			struct isert_device *device = isert_conn->conn_device;
> +			struct iscsi_conn *conn = isert_conn->conn;
> +			struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
> +
> +			if (cmd->i_state == ISTATE_SEND_LOGOUTRSP &&
> +			    conn->conn_state == TARG_CONN_STATE_IN_LOGOUT)
> +				isert_response_completion(desc, isert_cmd, isert_conn,
> +							  device->ib_device);
> +			else
> +				isert_completion_put(desc, isert_cmd, ib_dev, true);
> +		}
>   	} else {
>   		isert_conn->post_recv_buf_count--;
>   		if (!isert_conn->post_recv_buf_count)
>

Hey Nic,

Not sure if calling a full logout handler for error completion makes the
most sense here. if this is an error completion it means we are
either in session teardown or just about to start it. So simply
completing conn_logout_comp might be sufficient.

Anyways it might make sense to actually see this type of race happening
before adding this code.

I think this is better off as a standalone patch.

Sagi.

Re: [PATCH 1/3] iscsi-target: Convert iscsi_thread_set usage to kthread.h

[Sagi Grimberg]

On 3/21/2015 8:16 AM, Nicholas A. Bellinger wrote:
> From: Nicholas Bellinger <nab@linux-iscsi.org>
>
> This patch converts iscsi-target code to use modern kthread.h API
> callers for creating RX/TX threads for each new iscsi_conn descriptor,
> and releasing associated RX/TX threads during connection shutdown.
>
> This is done using iscsit_start_kthreads() -> kthread_run() to start
> new kthreads from within iscsi_post_login_handler(), and invoking
> kthread_stop() from existing iscsit_close_connection() code.
>
> Also, convert iscsit_logout_post_handler_closesession() code to use
> cmpxchg when determing when iscsit_cause_connection_reinstatement()
> needs to sleep waiting for completion.

I'll run some tests with this.

two comments below.

>
> Reported-by: Sagi Grimberg <sagig@mellanox.com>
> Cc: Sagi Grimberg <sagig@mellanox.com>
> Cc: Slava Shwartsman <valyushash@gmail.com>
> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
> ---
>   drivers/target/iscsi/iscsi_target.c       | 106 +++++++++++++-----------------
>   drivers/target/iscsi/iscsi_target_erl0.c  |  13 ++--
>   drivers/target/iscsi/iscsi_target_login.c |  59 +++++++++++++++--
>   include/target/iscsi/iscsi_target_core.h  |   7 ++
>   4 files changed, 116 insertions(+), 69 deletions(-)
>
> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
> index 2accb6e..cb97b59 100644
> --- a/drivers/target/iscsi/iscsi_target.c
> +++ b/drivers/target/iscsi/iscsi_target.c
> @@ -537,7 +537,7 @@ static struct iscsit_transport iscsi_target_transport = {
>
>   static int __init iscsi_target_init_module(void)
>   {
> -	int ret = 0;
> +	int ret = 0, size;
>
>   	pr_debug("iSCSI-Target "ISCSIT_VERSION"\n");
>
> @@ -546,6 +546,7 @@ static int __init iscsi_target_init_module(void)
>   		pr_err("Unable to allocate memory for iscsit_global\n");
>   		return -1;
>   	}
> +	spin_lock_init(&iscsit_global->ts_bitmap_lock);
>   	mutex_init(&auth_id_lock);
>   	spin_lock_init(&sess_idr_lock);
>   	idr_init(&tiqn_idr);
> @@ -555,15 +556,11 @@ static int __init iscsi_target_init_module(void)
>   	if (ret < 0)
>   		goto out;
>
> -	ret = iscsi_thread_set_init();
> -	if (ret < 0)
> +	size = BITS_TO_LONGS(ISCSIT_BITMAP_BITS) * sizeof(long);
> +	iscsit_global->ts_bitmap = vzalloc(size);
> +	if (!iscsit_global->ts_bitmap) {
> +		pr_err("Unable to allocate iscsit_global->ts_bitmap\n");
>   		goto configfs_out;
> -
> -	if (iscsi_allocate_thread_sets(TARGET_THREAD_SET_COUNT) !=
> -			TARGET_THREAD_SET_COUNT) {
> -		pr_err("iscsi_allocate_thread_sets() returned"
> -			" unexpected value!\n");
> -		goto ts_out1;
>   	}
>
>   	lio_qr_cache = kmem_cache_create("lio_qr_cache",
> @@ -572,7 +569,7 @@ static int __init iscsi_target_init_module(void)
>   	if (!lio_qr_cache) {
>   		pr_err("nable to kmem_cache_create() for"
>   				" lio_qr_cache\n");
> -		goto ts_out2;
> +		goto bitmap_out;
>   	}
>
>   	lio_dr_cache = kmem_cache_create("lio_dr_cache",
> @@ -617,10 +614,8 @@ dr_out:
>   	kmem_cache_destroy(lio_dr_cache);
>   qr_out:
>   	kmem_cache_destroy(lio_qr_cache);
> -ts_out2:
> -	iscsi_deallocate_thread_sets();
> -ts_out1:
> -	iscsi_thread_set_free();
> +bitmap_out:
> +	vfree(iscsit_global->ts_bitmap);
>   configfs_out:
>   	iscsi_target_deregister_configfs();
>   out:
> @@ -630,8 +625,6 @@ out:
>
>   static void __exit iscsi_target_cleanup_module(void)
>   {
> -	iscsi_deallocate_thread_sets();
> -	iscsi_thread_set_free();
>   	iscsit_release_discovery_tpg();
>   	iscsit_unregister_transport(&iscsi_target_transport);
>   	kmem_cache_destroy(lio_qr_cache);
> @@ -641,6 +634,7 @@ static void __exit iscsi_target_cleanup_module(void)
>
>   	iscsi_target_deregister_configfs();
>
> +	vfree(iscsit_global->ts_bitmap);
>   	kfree(iscsit_global);
>   }
>
> @@ -3710,17 +3704,16 @@ static int iscsit_send_reject(
>
>   void iscsit_thread_get_cpumask(struct iscsi_conn *conn)
>   {
> -	struct iscsi_thread_set *ts = conn->thread_set;
>   	int ord, cpu;
>   	/*
> -	 * thread_id is assigned from iscsit_global->ts_bitmap from
> -	 * within iscsi_thread_set.c:iscsi_allocate_thread_sets()
> +	 * bitmap_id is assigned from iscsit_global->ts_bitmap from
> +	 * within iscsit_start_kthreads()
>   	 *
> -	 * Here we use thread_id to determine which CPU that this
> -	 * iSCSI connection's iscsi_thread_set will be scheduled to
> +	 * Here we use bitmap_id to determine which CPU that this
> +	 * iSCSI connection's RX/TX threads will be scheduled to
>   	 * execute upon.
>   	 */
> -	ord = ts->thread_id % cpumask_weight(cpu_online_mask);
> +	ord = conn->bitmap_id % cpumask_weight(cpu_online_mask);
>   	for_each_online_cpu(cpu) {
>   		if (ord-- == 0) {
>   			cpumask_set_cpu(cpu, conn->conn_cpumask);
> @@ -3909,7 +3902,7 @@ check_rsp_state:
>   	switch (state) {
>   	case ISTATE_SEND_LOGOUTRSP:
>   		if (!iscsit_logout_post_handler(cmd, conn))
> -			goto restart;
> +			return -ECONNRESET;
>   		/* fall through */
>   	case ISTATE_SEND_STATUS:
>   	case ISTATE_SEND_ASYNCMSG:
> @@ -3937,8 +3930,6 @@ check_rsp_state:
>
>   err:
>   	return -1;
> -restart:
> -	return -EAGAIN;
>   }
>
>   static int iscsit_handle_response_queue(struct iscsi_conn *conn)
> @@ -3965,21 +3956,13 @@ static int iscsit_handle_response_queue(struct iscsi_conn *conn)
>   int iscsi_target_tx_thread(void *arg)
>   {
>   	int ret = 0;
> -	struct iscsi_conn *conn;
> -	struct iscsi_thread_set *ts = arg;
> +	struct iscsi_conn *conn = arg;
>   	/*
>   	 * Allow ourselves to be interrupted by SIGINT so that a
>   	 * connection recovery / failure event can be triggered externally.
>   	 */
>   	allow_signal(SIGINT);
>
> -restart:
> -	conn = iscsi_tx_thread_pre_handler(ts);
> -	if (!conn)
> -		goto out;
> -
> -	ret = 0;
> -
>   	while (!kthread_should_stop()) {
>   		/*
>   		 * Ensure that both TX and RX per connection kthreads
> @@ -3988,11 +3971,9 @@ restart:
>   		iscsit_thread_check_cpumask(conn, current, 1);
>
>   		wait_event_interruptible(conn->queues_wq,
> -					 !iscsit_conn_all_queues_empty(conn) ||
> -					 ts->status == ISCSI_THREAD_SET_RESET);
> +					 !iscsit_conn_all_queues_empty(conn));
>
> -		if ((ts->status == ISCSI_THREAD_SET_RESET) ||
> -		     signal_pending(current))
> +		if (signal_pending(current))
>   			goto transport_err;
>
>   get_immediate:
> @@ -4003,15 +3984,14 @@ get_immediate:
>   		ret = iscsit_handle_response_queue(conn);
>   		if (ret == 1)
>   			goto get_immediate;
> -		else if (ret == -EAGAIN)
> -			goto restart;
> +		else if (ret == -ECONNRESET)
> +			goto out;
>   		else if (ret < 0)
>   			goto transport_err;
>   	}
>
>   transport_err:
>   	iscsit_take_action_for_connection_exit(conn);
> -	goto restart;
>   out:
>   	return 0;
>   }
> @@ -4106,8 +4086,7 @@ int iscsi_target_rx_thread(void *arg)
>   	int ret;
>   	u8 buffer[ISCSI_HDR_LEN], opcode;
>   	u32 checksum = 0, digest = 0;
> -	struct iscsi_conn *conn = NULL;
> -	struct iscsi_thread_set *ts = arg;
> +	struct iscsi_conn *conn = arg;
>   	struct kvec iov;
>   	/*
>   	 * Allow ourselves to be interrupted by SIGINT so that a
> @@ -4115,11 +4094,6 @@ int iscsi_target_rx_thread(void *arg)
>   	 */
>   	allow_signal(SIGINT);
>
> -restart:
> -	conn = iscsi_rx_thread_pre_handler(ts);
> -	if (!conn)
> -		goto out;
> -
>   	if (conn->conn_transport->transport_type == ISCSI_INFINIBAND) {
>   		struct completion comp;
>   		int rc;
> @@ -4129,7 +4103,7 @@ restart:
>   		if (rc < 0)
>   			goto transport_err;
>
> -		goto out;
> +		goto transport_err;
>   	}
>
>   	while (!kthread_should_stop()) {
> @@ -4205,8 +4179,6 @@ transport_err:
>   	if (!signal_pending(current))
>   		atomic_set(&conn->transport_failed, 1);
>   	iscsit_take_action_for_connection_exit(conn);
> -	goto restart;
> -out:
>   	return 0;
>   }
>
> @@ -4268,7 +4240,26 @@ int iscsit_close_connection(
>   	if (conn->conn_transport->transport_type == ISCSI_TCP)
>   		complete(&conn->conn_logout_comp);
>
> -	iscsi_release_thread_set(conn);
> +	if (!strcmp(current->comm, ISCSI_RX_THREAD_NAME)) {
> +		if (conn->tx_thread &&
> +		    cmpxchg(&conn->tx_thread_active, true, false)) {
> +			printk("close_conn signal tx_thread\n");

This print needs to drop (or convert to pr_debug)

> +			send_sig(SIGINT, conn->tx_thread, 1);
> +			kthread_stop(conn->tx_thread);
> +		}
> +	} else if (!strcmp(current->comm, ISCSI_TX_THREAD_NAME)) {
> +		if (conn->rx_thread &&
> +		    cmpxchg(&conn->rx_thread_active, true, false)) {
> +			printk("close_conn signal rx_thread\n");

This print needs to drop (or convert to pr_debug)

> +			send_sig(SIGINT, conn->rx_thread, 1);
> +			kthread_stop(conn->rx_thread);
> +		}
> +	}
> +
> +	spin_lock(&iscsit_global->ts_bitmap_lock);
> +	bitmap_release_region(iscsit_global->ts_bitmap, conn->bitmap_id,
> +			      get_order(1));
> +	spin_unlock(&iscsit_global->ts_bitmap_lock);
>
>   	iscsit_stop_timers_for_cmds(conn);
>   	iscsit_stop_nopin_response_timer(conn);
> @@ -4546,15 +4537,13 @@ static void iscsit_logout_post_handler_closesession(
>   	struct iscsi_conn *conn)
>   {
>   	struct iscsi_session *sess = conn->sess;
> -
> -	iscsi_set_thread_clear(conn, ISCSI_CLEAR_TX_THREAD);
> -	iscsi_set_thread_set_signal(conn, ISCSI_SIGNAL_TX_THREAD);
> +	int sleep = cmpxchg(&conn->tx_thread_active, true, false);
>
>   	atomic_set(&conn->conn_logout_remove, 0);
>   	complete(&conn->conn_logout_comp);
>
>   	iscsit_dec_conn_usage_count(conn);
> -	iscsit_stop_session(sess, 1, 1);
> +	iscsit_stop_session(sess, sleep, sleep);
>   	iscsit_dec_session_usage_count(sess);
>   	target_put_session(sess->se_sess);
>   }
> @@ -4562,13 +4551,12 @@ static void iscsit_logout_post_handler_closesession(
>   static void iscsit_logout_post_handler_samecid(
>   	struct iscsi_conn *conn)
>   {
> -	iscsi_set_thread_clear(conn, ISCSI_CLEAR_TX_THREAD);
> -	iscsi_set_thread_set_signal(conn, ISCSI_SIGNAL_TX_THREAD);
> +	int sleep = cmpxchg(&conn->tx_thread_active, true, false);
>
>   	atomic_set(&conn->conn_logout_remove, 0);
>   	complete(&conn->conn_logout_comp);
>
> -	iscsit_cause_connection_reinstatement(conn, 1);
> +	iscsit_cause_connection_reinstatement(conn, sleep);
>   	iscsit_dec_conn_usage_count(conn);
>   }
>
> diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
> index bdd8731..e008ed2 100644
> --- a/drivers/target/iscsi/iscsi_target_erl0.c
> +++ b/drivers/target/iscsi/iscsi_target_erl0.c
> @@ -860,7 +860,10 @@ void iscsit_connection_reinstatement_rcfr(struct iscsi_conn *conn)
>   	}
>   	spin_unlock_bh(&conn->state_lock);
>
> -	iscsi_thread_set_force_reinstatement(conn);
> +	if (conn->tx_thread && conn->tx_thread_active)
> +		send_sig(SIGINT, conn->tx_thread, 1);
> +	if (conn->rx_thread && conn->rx_thread_active)
> +		send_sig(SIGINT, conn->rx_thread, 1);
>
>   sleep:
>   	wait_for_completion(&conn->conn_wait_rcfr_comp);
> @@ -885,10 +888,10 @@ void iscsit_cause_connection_reinstatement(struct iscsi_conn *conn, int sleep)
>   		return;
>   	}
>
> -	if (iscsi_thread_set_force_reinstatement(conn) < 0) {
> -		spin_unlock_bh(&conn->state_lock);
> -		return;
> -	}
> +	if (conn->tx_thread && conn->tx_thread_active)
> +		send_sig(SIGINT, conn->tx_thread, 1);
> +	if (conn->rx_thread && conn->rx_thread_active)
> +		send_sig(SIGINT, conn->rx_thread, 1);
>
>   	atomic_set(&conn->connection_reinstatement, 1);
>   	if (!sleep) {
> diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
> index 153fb66..345f073 100644
> --- a/drivers/target/iscsi/iscsi_target_login.c
> +++ b/drivers/target/iscsi/iscsi_target_login.c
> @@ -699,6 +699,51 @@ static void iscsi_post_login_start_timers(struct iscsi_conn *conn)
>   		iscsit_start_nopin_timer(conn);
>   }
>
> +int iscsit_start_kthreads(struct iscsi_conn *conn)
> +{
> +	int ret = 0;
> +
> +	spin_lock(&iscsit_global->ts_bitmap_lock);
> +	conn->bitmap_id = bitmap_find_free_region(iscsit_global->ts_bitmap,
> +					ISCSIT_BITMAP_BITS, get_order(1));
> +	spin_unlock(&iscsit_global->ts_bitmap_lock);
> +
> +	if (conn->bitmap_id < 0) {
> +		pr_err("bitmap_find_free_region() failed for"
> +		       " iscsit_start_kthreads()\n");
> +		return -ENOMEM;
> +	}
> +
> +	conn->tx_thread = kthread_run(iscsi_target_tx_thread, conn,
> +				      "%s", ISCSI_TX_THREAD_NAME);
> +	if (IS_ERR(conn->tx_thread)) {
> +		pr_err("Unable to start iscsi_target_tx_thread\n");
> +		ret = PTR_ERR(conn->tx_thread);
> +		goto out_bitmap;
> +	}
> +	conn->tx_thread_active = true;
> +
> +	conn->rx_thread = kthread_run(iscsi_target_rx_thread, conn,
> +				      "%s", ISCSI_RX_THREAD_NAME);
> +	if (IS_ERR(conn->rx_thread)) {
> +		pr_err("Unable to start iscsi_target_rx_thread\n");
> +		ret = PTR_ERR(conn->rx_thread);
> +		goto out_tx;
> +	}
> +	conn->rx_thread_active = true;
> +
> +	return 0;
> +out_tx:
> +	kthread_stop(conn->tx_thread);
> +	conn->tx_thread_active = false;
> +out_bitmap:
> +	spin_lock(&iscsit_global->ts_bitmap_lock);
> +	bitmap_release_region(iscsit_global->ts_bitmap, conn->bitmap_id,
> +			      get_order(1));
> +	spin_unlock(&iscsit_global->ts_bitmap_lock);
> +	return ret;
> +}
> +
>   int iscsi_post_login_handler(
>   	struct iscsi_np *np,
>   	struct iscsi_conn *conn,
> @@ -709,7 +754,7 @@ int iscsi_post_login_handler(
>   	struct se_session *se_sess = sess->se_sess;
>   	struct iscsi_portal_group *tpg = sess->tpg;
>   	struct se_portal_group *se_tpg = &tpg->tpg_se_tpg;
> -	struct iscsi_thread_set *ts;
> +	int rc;
>
>   	iscsit_inc_conn_usage_count(conn);
>
> @@ -724,7 +769,6 @@ int iscsi_post_login_handler(
>   	/*
>   	 * SCSI Initiator -> SCSI Target Port Mapping
>   	 */
> -	ts = iscsi_get_thread_set();
>   	if (!zero_tsih) {
>   		iscsi_set_session_parameters(sess->sess_ops,
>   				conn->param_list, 0);
> @@ -751,9 +795,11 @@ int iscsi_post_login_handler(
>   			sess->sess_ops->InitiatorName);
>   		spin_unlock_bh(&sess->conn_lock);
>
> -		iscsi_post_login_start_timers(conn);
> +		rc = iscsit_start_kthreads(conn);
> +		if (rc)
> +			return rc;
>
> -		iscsi_activate_thread_set(conn, ts);
> +		iscsi_post_login_start_timers(conn);
>   		/*
>   		 * Determine CPU mask to ensure connection's RX and TX kthreads
>   		 * are scheduled on the same CPU.
> @@ -810,8 +856,11 @@ int iscsi_post_login_handler(
>   		" iSCSI Target Portal Group: %hu\n", tpg->nsessions, tpg->tpgt);
>   	spin_unlock_bh(&se_tpg->session_lock);
>
> +	rc = iscsit_start_kthreads(conn);
> +	if (rc)
> +		return rc;
> +
>   	iscsi_post_login_start_timers(conn);
> -	iscsi_activate_thread_set(conn, ts);
>   	/*
>   	 * Determine CPU mask to ensure connection's RX and TX kthreads
>   	 * are scheduled on the same CPU.
> diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h
> index d3583d3..dd0f3ab 100644
> --- a/include/target/iscsi/iscsi_target_core.h
> +++ b/include/target/iscsi/iscsi_target_core.h
> @@ -602,6 +602,11 @@ struct iscsi_conn {
>   	struct iscsi_session	*sess;
>   	/* Pointer to thread_set in use for this conn's threads */
>   	struct iscsi_thread_set	*thread_set;
> +	int			bitmap_id;
> +	int			rx_thread_active;
> +	struct task_struct	*rx_thread;
> +	int			tx_thread_active;
> +	struct task_struct	*tx_thread;
>   	/* list_head for session connection list */
>   	struct list_head	conn_list;
>   } ____cacheline_aligned;
> @@ -871,10 +876,12 @@ struct iscsit_global {
>   	/* Unique identifier used for the authentication daemon */
>   	u32			auth_id;
>   	u32			inactive_ts;
> +#define ISCSIT_BITMAP_BITS	262144
>   	/* Thread Set bitmap count */
>   	int			ts_bitmap_count;
>   	/* Thread Set bitmap pointer */
>   	unsigned long		*ts_bitmap;
> +	spinlock_t		ts_bitmap_lock;
>   	/* Used for iSCSI discovery session authentication */
>   	struct iscsi_node_acl	discovery_acl;
>   	struct iscsi_portal_group	*discovery_tpg;
>

Re: [PATCH 1/3] iscsi-target: Convert iscsi_thread_set usage to kthread.h

[Sagi Grimberg]

On 3/23/2015 2:21 PM, Sagi Grimberg wrote:
> On 3/21/2015 8:16 AM, Nicholas A. Bellinger wrote:
>> From: Nicholas Bellinger <nab@linux-iscsi.org>
>>
>> This patch converts iscsi-target code to use modern kthread.h API
>> callers for creating RX/TX threads for each new iscsi_conn descriptor,
>> and releasing associated RX/TX threads during connection shutdown.
>>
>> This is done using iscsit_start_kthreads() -> kthread_run() to start
>> new kthreads from within iscsi_post_login_handler(), and invoking
>> kthread_stop() from existing iscsit_close_connection() code.
>>
>> Also, convert iscsit_logout_post_handler_closesession() code to use
>> cmpxchg when determing when iscsit_cause_connection_reinstatement()
>> needs to sleep waiting for completion.
>
> I'll run some tests with this.

Hi Nic,

I can report that this series (minus patch 3/3 and along with some
patches I'll send out soon) seems to resolve the list corruption issue
I've seen.

Overall the patches look good to me so feel free to include
patches 1+2 with:

Tested-by: Sagi Grimberg <sagig@mellanox.com>

Thanks!
Sagi.

Re: [PATCH 1/3] iscsi-target: Convert iscsi_thread_set usage to kthread.h

[Nicholas A. Bellinger]

On Tue, 2015-03-24 at 18:37 +0200, Sagi Grimberg wrote:
> On 3/23/2015 2:21 PM, Sagi Grimberg wrote:
> > On 3/21/2015 8:16 AM, Nicholas A. Bellinger wrote:
> >> From: Nicholas Bellinger <nab@linux-iscsi.org>
> >>
> >> This patch converts iscsi-target code to use modern kthread.h API
> >> callers for creating RX/TX threads for each new iscsi_conn descriptor,
> >> and releasing associated RX/TX threads during connection shutdown.
> >>
> >> This is done using iscsit_start_kthreads() -> kthread_run() to start
> >> new kthreads from within iscsi_post_login_handler(), and invoking
> >> kthread_stop() from existing iscsit_close_connection() code.
> >>
> >> Also, convert iscsit_logout_post_handler_closesession() code to use
> >> cmpxchg when determing when iscsit_cause_connection_reinstatement()
> >> needs to sleep waiting for completion.
> >
> > I'll run some tests with this.
> 
> Hi Nic,
> 
> I can report that this series (minus patch 3/3 and along with some
> patches I'll send out soon) seems to resolve the list corruption issue
> I've seen.
> 
> Overall the patches look good to me so feel free to include
> patches 1+2 with:
> 
> Tested-by: Sagi Grimberg <sagig@mellanox.com>
> 

Great.  The first two are queued in for-next, and for the moment #1 has
a v3.10+ stable tag as well..

Please let me know if you run into any regressions with these.

Thanks alot Sagi!

--nab