[BUG] iscsi hangs during login attempt

* [BUG] iscsi hangs during login attempt
@ 2022-09-14 18:00 Paul Dagnelie
  2022-09-19 17:01 ` Maurizio Lombardi
  2022-09-20 15:49 ` Maurizio Lombardi
  0 siblings, 2 replies; 7+ messages in thread
From: Paul Dagnelie @ 2022-09-14 18:00 UTC (permalink / raw)
  To: target-devel; +Cc: Sumedh Bala, Sebastien Roy

Hi all,

We've encountered an issue a few times now that results in a hung
iscsi session that cannot be recovered, and ultimately leads to the
system being wedged. I've done some initial analysis on what the issue
might be, but I would appreciate some more experienced eyes on it and
suggestions about the right course of action.

The machines in question are running Linux 5.4 (plus some other
changes; it's a custom kernel built on top of Ubuntu 20.04, but I
don't see any relevant patches anywhere in the stack, so I decided to
come straight here). The issue presents with iscsi login attempts
timing out repeatedly. We took a core dump of the machine, and
analysis of the stack traces showed that the iscsi_target_login_thread
is waiting for another thread to finish with the np_login_sem. That
semaphore is being held by a thread in iscsi_target_do_login_rx, which
is currently waiting for a session to be reinstated.
iscsit_cause_connection_reinstatement signals the rx and tx threads,
and then waits for the conn_wait_comp completion, which is signalled
in iscsit_close_connection. That appears to be called by the tx and rx
threads when they exit.  After puzzling through the core for a bit, I
found the kernel threads in question, and they appear to be calmly
waiting in the normal blocking path waiting for IOs to come in for
them to respond to. I would think that if they were in that state when
the SIGINT came in they would have exited properly.

My theory, after examining the code, is if two connection requests
were received from one initiator in rapid succession, it seems like
the second one would use the connection reinstatement logic. It may be
possible that if a reinstatement happens fast enough after the initial
login, the rx and tx threads would not yet have marked themselves as
able to receive the SIGINT that the connection reinstatement logic
uses to prompt them to close the connection so a new one can be
created. As I understand how signals are processed for kernel threads,
this would result in the signal being dropped. After that happens, the
kernel threads would never exit. If that is possible, then this could
explain the issue we’re seeing.

We've enabled some of the pr_debug statements on the systems in
question, but the issue hasn't recurred on any system yet. In
addition, it's worth noting that this issue did present itself shortly
after patching the initiating windows systems, which would presumably
result in one or more connection reinstatements. Does this theory seem
plausible? We haven't managed to reproduce it in-house or with
debugging statements enabled yet, but if it is the root cause it seems
to me the best fix would be to add or use an existing flag that is set
during reconnection (before the signal is sent), and have the rx and
tx threads check it after enabling signals to close the window for the
race.

-- 
Paul Dagnelie

^ permalink raw reply	[flat|nested] 7+ messages in thread