* Re: [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem
2020-07-29 13:03 [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem Hou Pu
@ 2020-08-31 12:02 ` Hou Pu
2020-09-02 2:57 ` Michael Christie
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Hou Pu @ 2020-08-31 12:02 UTC (permalink / raw)
To: martin.petersen, linux-scsi, target-devel, michael.christie; +Cc: stable
Hi,
Could anyone help review this patch? It is an important
fix.
Login thread could hang __forever__ and only reboot could
solve this. This happened several times in our production
environment.
np->np_login_timer is shared by all TPGs of a portal.
should be used carefully. If a connection to iqn-TPG A start
login timer, It should not stop by another connection
to iqn-TPG B.
np->np_login_timer protect potential hangs in
__iscsi_target_login_thread.
should be used locally in this function. It should really
not be used in iscsi_target_login_sess_out from workqueue
context.
Thanks,
Hou
On 2020/7/29 9:03 PM, Hou Pu wrote:
> The iscsi target login thread might stuck in following stack:
>
> cat /proc/`pidof iscsi_np`/stack
> [<0>] down_interruptible+0x42/0x50
> [<0>] iscsit_access_np+0xe3/0x167
> [<0>] iscsi_target_locate_portal+0x695/0x8ac
> [<0>] __iscsi_target_login_thread+0x855/0xb82
> [<0>] iscsi_target_login_thread+0x2f/0x5a
> [<0>] kthread+0xfa/0x130
> [<0>] ret_from_fork+0x1f/0x30
>
> This could be reproduced by following steps:
> 1. Initiator A try to login iqn1-tpg1 on port 3260. After finishing
> PDU exchange in the login thread and before the negotiation is
> finished, at this time the network link is down. In a production
> environment, this could happen. I could emulated it by bring
> the network card down in the initiator node by ifconfig eth0 down.
> (Now A could never finish this login. And tpg->np_login_sem is
> hold by it).
> 2. Initiator B try to login iqn2-tpg1 on port 3260. After finishing
> PDU exchange in the login thread. The target expect to process
> remaining login PDUs in workqueue context.
> 3. Initiator A' try to re-login to iqn1-tpg1 on port 3260 from
> a new socket. It will wait for tpg->np_login_sem with
> np->np_login_timer loaded to wait for at most 15 second.
> (Because the lock is held by A. A never gets a change to
> release tpg->np_login_sem. so A' should finally get timeout).
> 4. Before A' got timeout. Initiator B gets negotiation failed and
> calls iscsi_target_login_drop()->iscsi_target_login_sess_out().
> The np->np_login_timer is canceled. And initiator A' will hang
> there forever. Because A' is now in the login thread. All other
> login requests could not be serviced.
>
> Fix this by moving iscsi_stop_login_thread_timer() out of
> iscsi_target_login_sess_out(). Also remove iscsi_np parameter
> from iscsi_target_login_sess_out().
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Hou Pu <houpu@bytedance.com>
> ---
> drivers/target/iscsi/iscsi_target_login.c | 6 +++---
> drivers/target/iscsi/iscsi_target_login.h | 3 +--
> drivers/target/iscsi/iscsi_target_nego.c | 3 +--
> 3 files changed, 5 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
> index 85748e338858..893d1b406c29 100644
> --- a/drivers/target/iscsi/iscsi_target_login.c
> +++ b/drivers/target/iscsi/iscsi_target_login.c
> @@ -1149,7 +1149,7 @@ void iscsit_free_conn(struct iscsi_conn *conn)
> }
>
> void iscsi_target_login_sess_out(struct iscsi_conn *conn,
> - struct iscsi_np *np, bool zero_tsih, bool new_sess)
> + bool zero_tsih, bool new_sess)
> {
> if (!new_sess)
> goto old_sess_out;
> @@ -1167,7 +1167,6 @@ void iscsi_target_login_sess_out(struct iscsi_conn *conn,
> conn->sess = NULL;
>
> old_sess_out:
> - iscsi_stop_login_thread_timer(np);
> /*
> * If login negotiation fails check if the Time2Retain timer
> * needs to be restarted.
> @@ -1407,8 +1406,9 @@ static int __iscsi_target_login_thread(struct iscsi_np *np)
> new_sess_out:
> new_sess = true;
> old_sess_out:
> + iscsi_stop_login_thread_timer(np);
> tpg_np = conn->tpg_np;
> - iscsi_target_login_sess_out(conn, np, zero_tsih, new_sess);
> + iscsi_target_login_sess_out(conn, zero_tsih, new_sess);
> new_sess = false;
>
> if (tpg) {
> diff --git a/drivers/target/iscsi/iscsi_target_login.h b/drivers/target/iscsi/iscsi_target_login.h
> index 3b8e3639ff5d..fc95e6150253 100644
> --- a/drivers/target/iscsi/iscsi_target_login.h
> +++ b/drivers/target/iscsi/iscsi_target_login.h
> @@ -22,8 +22,7 @@ extern int iscsit_put_login_tx(struct iscsi_conn *, struct iscsi_login *, u32);
> extern void iscsit_free_conn(struct iscsi_conn *);
> extern int iscsit_start_kthreads(struct iscsi_conn *);
> extern void iscsi_post_login_handler(struct iscsi_np *, struct iscsi_conn *, u8);
> -extern void iscsi_target_login_sess_out(struct iscsi_conn *, struct iscsi_np *,
> - bool, bool);
> +extern void iscsi_target_login_sess_out(struct iscsi_conn *, bool, bool);
> extern int iscsi_target_login_thread(void *);
> extern void iscsi_handle_login_thread_timeout(struct timer_list *t);
>
> diff --git a/drivers/target/iscsi/iscsi_target_nego.c b/drivers/target/iscsi/iscsi_target_nego.c
> index 685d771b51d4..e32d93b92742 100644
> --- a/drivers/target/iscsi/iscsi_target_nego.c
> +++ b/drivers/target/iscsi/iscsi_target_nego.c
> @@ -535,12 +535,11 @@ static bool iscsi_target_sk_check_and_clear(struct iscsi_conn *conn, unsigned in
>
> static void iscsi_target_login_drop(struct iscsi_conn *conn, struct iscsi_login *login)
> {
> - struct iscsi_np *np = login->np;
> bool zero_tsih = login->zero_tsih;
>
> iscsi_remove_failed_auth_entry(conn);
> iscsi_target_nego_release(conn);
> - iscsi_target_login_sess_out(conn, np, zero_tsih, true);
> + iscsi_target_login_sess_out(conn, zero_tsih, true);
> }
>
> struct conn_timeout {
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem
2020-07-29 13:03 [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem Hou Pu
2020-08-31 12:02 ` Hou Pu
@ 2020-09-02 2:57 ` Michael Christie
2020-09-02 4:16 ` Hou Pu
2020-09-02 19:02 ` Mike Christie
2020-09-03 3:00 ` Martin K. Petersen
3 siblings, 1 reply; 6+ messages in thread
From: Michael Christie @ 2020-09-02 2:57 UTC (permalink / raw)
To: Hou Pu; +Cc: Martin K. Petersen, linux-scsi, target-devel, stable
> On Jul 29, 2020, at 8:03 AM, Hou Pu <houpu@bytedance.com> wrote:
>
> The iscsi target login thread might stuck in following stack:
>
> cat /proc/`pidof iscsi_np`/stack
> [<0>] down_interruptible+0x42/0x50
> [<0>] iscsit_access_np+0xe3/0x167
> [<0>] iscsi_target_locate_portal+0x695/0x8ac
> [<0>] __iscsi_target_login_thread+0x855/0xb82
> [<0>] iscsi_target_login_thread+0x2f/0x5a
> [<0>] kthread+0xfa/0x130
> [<0>] ret_from_fork+0x1f/0x30
>
> This could be reproduced by following steps:
> 1. Initiator A try to login iqn1-tpg1 on port 3260. After finishing
> PDU exchange in the login thread and before the negotiation is
> finished, at this time the network link is down. In a production
> environment, this could happen. I could emulated it by bring
> the network card down in the initiator node by ifconfig eth0 down.
> (Now A could never finish this login. And tpg->np_login_sem is
> hold by it).
> 2. Initiator B try to login iqn2-tpg1 on port 3260. After finishing
> PDU exchange in the login thread. The target expect to process
> remaining login PDUs in workqueue context.
> 3. Initiator A' try to re-login to iqn1-tpg1 on port 3260 from
> a new socket. It will wait for tpg->np_login_sem with
> np->np_login_timer loaded to wait for at most 15 second.
> (Because the lock is held by A. A never gets a change to
> release tpg->np_login_sem. so A' should finally get timeout).
> 4. Before A' got timeout. Initiator B gets negotiation failed and
> calls iscsi_target_login_drop()->iscsi_target_login_sess_out().
> The np->np_login_timer is canceled. And initiator A' will hang
> there forever. Because A' is now in the login thread. All other
> login requests could not be serviced.
iqn1 and iqn1 are different targets right? It’s not clear to me how when initiator B fails negotiation that it cancels the timer for the portal under a different iqn/target.
Is iqn2-tpg1->np1 a different struct than iqn1-tpg1-np1? I mean iscsit_get_tpg_from_np would return a different np struct for initiator B and for A?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem
2020-09-02 2:57 ` Michael Christie
@ 2020-09-02 4:16 ` Hou Pu
0 siblings, 0 replies; 6+ messages in thread
From: Hou Pu @ 2020-09-02 4:16 UTC (permalink / raw)
To: Michael Christie; +Cc: Martin K. Petersen, linux-scsi, target-devel, stable
On 2020/9/2 10:57 AM, Michael Christie wrote:
>
>
>> On Jul 29, 2020, at 8:03 AM, Hou Pu <houpu@bytedance.com> wrote:
>>
>> The iscsi target login thread might stuck in following stack:
>>
>> cat /proc/`pidof iscsi_np`/stack
>> [<0>] down_interruptible+0x42/0x50
>> [<0>] iscsit_access_np+0xe3/0x167
>> [<0>] iscsi_target_locate_portal+0x695/0x8ac
>> [<0>] __iscsi_target_login_thread+0x855/0xb82
>> [<0>] iscsi_target_login_thread+0x2f/0x5a
>> [<0>] kthread+0xfa/0x130
>> [<0>] ret_from_fork+0x1f/0x30
>>
>> This could be reproduced by following steps:
>> 1. Initiator A try to login iqn1-tpg1 on port 3260. After finishing
>> PDU exchange in the login thread and before the negotiation is
>> finished, at this time the network link is down. In a production
>> environment, this could happen. I could emulated it by bring
>> the network card down in the initiator node by ifconfig eth0 down.
>> (Now A could never finish this login. And tpg->np_login_sem is
>> hold by it).
>> 2. Initiator B try to login iqn2-tpg1 on port 3260. After finishing
>> PDU exchange in the login thread. The target expect to process
>> remaining login PDUs in workqueue context.
>> 3. Initiator A' try to re-login to iqn1-tpg1 on port 3260 from
>> a new socket. It will wait for tpg->np_login_sem with
>> np->np_login_timer loaded to wait for at most 15 second.
>> (Because the lock is held by A. A never gets a change to
>> release tpg->np_login_sem. so A' should finally get timeout).
>> 4. Before A' got timeout. Initiator B gets negotiation failed and
>> calls iscsi_target_login_drop()->iscsi_target_login_sess_out().
>> The np->np_login_timer is canceled. And initiator A' will hang
>> there forever. Because A' is now in the login thread. All other
>> login requests could not be serviced.
>
> iqn1 and iqn1 are different targets right? It’s not clear to me how when initiator B fails negotiation that it cancels the timer for the portal under a different iqn/target.
iqn1-tpg1 in step1 and step3 are same one. (same target volume)
iqn2-tpg1 in step2 is a different volume on the same host.
The configuration likes below:
iqn1-tpg1:
root@storageXXX:/sys/kernel/config/target/iscsi# ls
iqn.2010-10.org.openstack\:volume-00e50deb-5296-4f18-xxxx-106f96a880c8/tpgt_1/np/
10.129.77.16:3260
iqn2-tpg1:
root@storageXXX:/sys/kernel/config/target/iscsi# ls
iqn.2010-10.org.openstack\:volume-86af15c6-c529-4715-xxxx-3c9ca068635d/tpgt_1/np/
10.129.77.16:3260
(I could provide more is needed)
>
> Is iqn2-tpg1->np1 a different struct than iqn1-tpg1-np1? I mean iscsit_get_tpg_from_np would return a different np struct for initiator B and for A?
>
iscsit_get_tpg_from_np() returned different struct iscsi_portal_group
for initiator A and B. But struct iscsi_np is shared by them.
Because they have the same portal(ip address and port).
Thanks,
Hou
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem
2020-07-29 13:03 [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem Hou Pu
2020-08-31 12:02 ` Hou Pu
2020-09-02 2:57 ` Michael Christie
@ 2020-09-02 19:02 ` Mike Christie
2020-09-03 3:00 ` Martin K. Petersen
3 siblings, 0 replies; 6+ messages in thread
From: Mike Christie @ 2020-09-02 19:02 UTC (permalink / raw)
To: Hou Pu, martin.petersen, linux-scsi, target-devel; +Cc: stable
On 7/29/20 8:03 AM, Hou Pu wrote:
> The iscsi target login thread might stuck in following stack:
>
> cat /proc/`pidof iscsi_np`/stack
> [<0>] down_interruptible+0x42/0x50
> [<0>] iscsit_access_np+0xe3/0x167
> [<0>] iscsi_target_locate_portal+0x695/0x8ac
> [<0>] __iscsi_target_login_thread+0x855/0xb82
> [<0>] iscsi_target_login_thread+0x2f/0x5a
> [<0>] kthread+0xfa/0x130
> [<0>] ret_from_fork+0x1f/0x30
>
> This could be reproduced by following steps:
> 1. Initiator A try to login iqn1-tpg1 on port 3260. After finishing
> PDU exchange in the login thread and before the negotiation is
> finished, at this time the network link is down. In a production
> environment, this could happen. I could emulated it by bring
> the network card down in the initiator node by ifconfig eth0 down.
> (Now A could never finish this login. And tpg->np_login_sem is
> hold by it).
> 2. Initiator B try to login iqn2-tpg1 on port 3260. After finishing
> PDU exchange in the login thread. The target expect to process
> remaining login PDUs in workqueue context.
> 3. Initiator A' try to re-login to iqn1-tpg1 on port 3260 from
> a new socket. It will wait for tpg->np_login_sem with
> np->np_login_timer loaded to wait for at most 15 second.
> (Because the lock is held by A. A never gets a change to
> release tpg->np_login_sem. so A' should finally get timeout).
> 4. Before A' got timeout. Initiator B gets negotiation failed and
> calls iscsi_target_login_drop()->iscsi_target_login_sess_out().
> The np->np_login_timer is canceled. And initiator A' will hang
> there forever. Because A' is now in the login thread. All other
> login requests could not be serviced.
>
> Fix this by moving iscsi_stop_login_thread_timer() out of
> iscsi_target_login_sess_out(). Also remove iscsi_np parameter
> from iscsi_target_login_sess_out().
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Hou Pu <houpu@bytedance.com>
> ---
> drivers/target/iscsi/iscsi_target_login.c | 6 +++---
> drivers/target/iscsi/iscsi_target_login.h | 3 +--
> drivers/target/iscsi/iscsi_target_nego.c | 3 +--
> 3 files changed, 5 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
> index 85748e338858..893d1b406c29 100644
> --- a/drivers/target/iscsi/iscsi_target_login.c
> +++ b/drivers/target/iscsi/iscsi_target_login.c
> @@ -1149,7 +1149,7 @@ void iscsit_free_conn(struct iscsi_conn *conn)
> }
>
> void iscsi_target_login_sess_out(struct iscsi_conn *conn,
> - struct iscsi_np *np, bool zero_tsih, bool new_sess)
> + bool zero_tsih, bool new_sess)
> {
> if (!new_sess)
> goto old_sess_out;
> @@ -1167,7 +1167,6 @@ void iscsi_target_login_sess_out(struct iscsi_conn *conn,
> conn->sess = NULL;
>
> old_sess_out:
> - iscsi_stop_login_thread_timer(np);
> /*
> * If login negotiation fails check if the Time2Retain timer
> * needs to be restarted.
> @@ -1407,8 +1406,9 @@ static int __iscsi_target_login_thread(struct iscsi_np *np)
> new_sess_out:
> new_sess = true;
> old_sess_out:
> + iscsi_stop_login_thread_timer(np);
> tpg_np = conn->tpg_np;
> - iscsi_target_login_sess_out(conn, np, zero_tsih, new_sess);
> + iscsi_target_login_sess_out(conn, zero_tsih, new_sess);
> new_sess = false;
>
> if (tpg) {
> diff --git a/drivers/target/iscsi/iscsi_target_login.h b/drivers/target/iscsi/iscsi_target_login.h
> index 3b8e3639ff5d..fc95e6150253 100644
> --- a/drivers/target/iscsi/iscsi_target_login.h
> +++ b/drivers/target/iscsi/iscsi_target_login.h
> @@ -22,8 +22,7 @@ extern int iscsit_put_login_tx(struct iscsi_conn *, struct iscsi_login *, u32);
> extern void iscsit_free_conn(struct iscsi_conn *);
> extern int iscsit_start_kthreads(struct iscsi_conn *);
> extern void iscsi_post_login_handler(struct iscsi_np *, struct iscsi_conn *, u8);
> -extern void iscsi_target_login_sess_out(struct iscsi_conn *, struct iscsi_np *,
> - bool, bool);
> +extern void iscsi_target_login_sess_out(struct iscsi_conn *, bool, bool);
> extern int iscsi_target_login_thread(void *);
> extern void iscsi_handle_login_thread_timeout(struct timer_list *t);
>
> diff --git a/drivers/target/iscsi/iscsi_target_nego.c b/drivers/target/iscsi/iscsi_target_nego.c
> index 685d771b51d4..e32d93b92742 100644
> --- a/drivers/target/iscsi/iscsi_target_nego.c
> +++ b/drivers/target/iscsi/iscsi_target_nego.c
> @@ -535,12 +535,11 @@ static bool iscsi_target_sk_check_and_clear(struct iscsi_conn *conn, unsigned in
>
> static void iscsi_target_login_drop(struct iscsi_conn *conn, struct iscsi_login *login)
> {
> - struct iscsi_np *np = login->np;
> bool zero_tsih = login->zero_tsih;
>
> iscsi_remove_failed_auth_entry(conn);
> iscsi_target_nego_release(conn);
> - iscsi_target_login_sess_out(conn, np, zero_tsih, true);
> + iscsi_target_login_sess_out(conn, zero_tsih, true);
> }
>
> struct conn_timeout {
>
Reviewed-by: Mike Christie <michael.christie@oracle.com>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem
2020-07-29 13:03 [PATCH] iscsi-target: fix hang in iscsit_access_np() when getting tpg->np_login_sem Hou Pu
` (2 preceding siblings ...)
2020-09-02 19:02 ` Mike Christie
@ 2020-09-03 3:00 ` Martin K. Petersen
3 siblings, 0 replies; 6+ messages in thread
From: Martin K. Petersen @ 2020-09-03 3:00 UTC (permalink / raw)
To: Hou Pu, michael.christie, target-devel, linux-scsi
Cc: Martin K . Petersen, stable
On Wed, 29 Jul 2020 09:03:43 -0400, Hou Pu wrote:
> The iscsi target login thread might stuck in following stack:
>
> cat /proc/`pidof iscsi_np`/stack
> [<0>] down_interruptible+0x42/0x50
> [<0>] iscsit_access_np+0xe3/0x167
> [<0>] iscsi_target_locate_portal+0x695/0x8ac
> [<0>] __iscsi_target_login_thread+0x855/0xb82
> [<0>] iscsi_target_login_thread+0x2f/0x5a
> [<0>] kthread+0xfa/0x130
> [<0>] ret_from_fork+0x1f/0x30
>
> [...]
Applied to 5.9/scsi-fixes, thanks!
[1/1] scsi: target: iscsi: Fix hang in iscsit_access_np() when getting tpg->np_login_sem
https://git.kernel.org/mkp/scsi/c/ed43ffea78dc
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 6+ messages in thread