[Bug 85751] New: iSCSI initiator lockup during login

[Bug 85751] New: iSCSI initiator lockup during login

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

            Bug ID: 85751
           Summary: iSCSI initiator lockup during login
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.16.4
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: linux-scsi@vger.kernel.org
          Reporter: bvanassche@acm.org
        Regression: No

Created attachment 152781
  --> https://bugzilla.kernel.org/attachment.cgi?id=152781&action=edit
Output generated by echo w > /proc/sysrq

If I interpret the sysrq output correctly this lockup is caused by the thread
with PID 15219 waiting inside async_synchronize_cookie_domain() while the
thread with PID 15221 is waiting for the ihost mutex held by the first thread ?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 85751] iSCSI initiator lockup during login

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Sagi Grimberg <sagig@dev.mellanox.co.il> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sagig@dev.mellanox.co.il

--- Comment #1 from Sagi Grimberg <sagig@dev.mellanox.co.il> ---
Hey Bart,

Any chance you provide on the scenario you used to reproduce this issue?
did you simply login and got it?
is it a firm reproduce?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 85751] iSCSI initiator lockup during logout

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Bart Van Assche <bvanassche@acm.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|iSCSI initiator lockup      |iSCSI initiator lockup
                   |during login                |during logout

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 85751] iSCSI initiator lockup during logout

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

--- Comment #2 from Bart Van Assche <bvanassche@acm.org> ---
This is not easy to reproduce and happened while testing a modification of
iSCSI target software. The following commands were issued at the initiator
side:
* iscsiadm -m discovery -p 192.168.1.124 -t st
* iscsiadm -m node -p 192.168.1.124 -l
(something went wrong at the target side and the iSCSI initiator tried to
relogin)
* iscsiadm -m node -p 192.168.1.124 -u
Even after I powered off the virtual machine that was running the iSCSI target
software that last logout command did not finish. The system load raised to a
high value due to several kernel threads at the initiator side that got stuck
in an uninterruptible wait. It is only after the logout command had been issued
that I ran "echo w > /proc/sysrq-trigger".

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 85751] iSCSI initiator lockup during logout

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Bart Van Assche <bvanassche@acm.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #152781|Output generated by echo w  |Output generated by echo w
        description|> /proc/sysrq               |> /proc/sysrq-trigger

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [Bug 85751] iSCSI initiator lockup during logout

[Mike Christie]

On 10/07/2014 11:42 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
> * iscsiadm -m discovery -p 192.168.1.124 -t st
> * iscsiadm -m node -p 192.168.1.124 -l
> (something went wrong at the target side and the iSCSI initiator tried to
> relogin)

Did this login command ever complete normally? Did you ctrl-c it or did
you run the logout command below in another terminal?

> * iscsiadm -m node -p 192.168.1.124 -u

The login command looks like it is stuck waiting for scan/setup related
commands to complete:

 [<ffffffff811f54f5>] blk_execute_rq+0x75/0xf0
 [<ffffffff811f16fa>] ? blk_rq_bio_prep+0x6a/0x90
 [<ffffffff811f52b9>] ? blk_rq_map_kern+0xb9/0x160
 [<ffffffff811eef0a>] ? blk_get_request+0x7a/0xc0
 [<ffffffffa0008e3b>] scsi_execute+0xdb/0x160 [scsi_mod]
 [<ffffffffa000a52e>] scsi_execute_req_flags+0x8e/0x100 [scsi_mod]
 [<ffffffffa000c8ed>] scsi_probe_and_add_lun+0x20d/0xc40 [scsi_mod]
 [<ffffffffa0008e69>] ? scsi_execute+0x109/0x160 [scsi_mod]
 [<ffffffffa000db02>] __scsi_scan_target+0x522/0x6c0 [scsi_mod]
 [<ffffffff812e040d>] ? __pm_runtime_resume+0x4d/0x60
 [<ffffffffa000dd90>] scsi_scan_target+0xf0/0x110 [scsi_mod]
 [<ffffffffa0852d6f>] iscsi_user_scan_session.part.13+0x10f/0x150
[scsi_transport_iscsi]
 [<ffffffffa0852db0>] ? iscsi_user_scan_session.part.13+0x150/0x150
[scsi_transport_iscsi]
 [<ffffffffa0852dd1>] iscsi_user_scan_session+0x21/0x30
[scsi_transport_iscsi]


And so above it should have the ihost->mutex held while stuck.

But here it looks the logout command has been run and we have passed the
part where we take the mutex in there and already started to delete the
target.


 [<ffffffff812d41c9>] device_del+0x129/0x1c0
 [<ffffffffa0010005>] __scsi_remove_device+0xd5/0xe0 [scsi_mod]
 [<ffffffffa0010036>] scsi_remove_device+0x26/0x40 [scsi_mod]
 [<ffffffffa00101f0>] scsi_remove_target+0x170/0x230 [scsi_mod]
 [<ffffffffa0851f84>] __iscsi_unbind_session+0xc4/0x1a0
[scsi_transport_iscsi]


Before you ran the logout command, did the login/scan hang? Did we ever
log back in?

Then did you run the logout command to try and clean things up.

Re: [Bug 85751] iSCSI initiator lockup during logout

[Bart Van Assche]

On 10/07/14 19:03, Mike Christie wrote:
> On 10/07/2014 11:42 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
>> * iscsiadm -m discovery -p 192.168.1.124 -t st
>> * iscsiadm -m node -p 192.168.1.124 -l
>> (something went wrong at the target side and the iSCSI initiator tried to
>> relogin)
>
> Did this login command ever complete normally? Did you ctrl-c it or did
> you run the logout command below in another terminal?

Hello Mike,

Thanks for jumping in. I had indeed interrupted login via Ctrl-C because 
login took longer than expected.

Bart.

[Bug 85751] iSCSI initiator lockup during logout

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

--- Comment #3 from Bart Van Assche <bvanassche@acm.org> ---
On 10/07/14 19:03, Mike Christie wrote:
> On 10/07/2014 11:42 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
>> * iscsiadm -m discovery -p 192.168.1.124 -t st
>> * iscsiadm -m node -p 192.168.1.124 -l
>> (something went wrong at the target side and the iSCSI initiator tried to
>> relogin)
>
> Did this login command ever complete normally? Did you ctrl-c it or did
> you run the logout command below in another terminal?

Hello Mike,

Thanks for jumping in. I had indeed interrupted login via Ctrl-C because 
login took longer than expected.

Bart.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 85751] iSCSI initiator lockup during logout

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Jaden <jaden1q84@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jaden1q84@gmail.com

--- Comment #4 from Jaden <jaden1q84@gmail.com> ---
I also encountered the similar issue, but not in logout stage. If the links are
down could also meet this issue occasionally. Below are my reproduce steps:

1.while :; do dd if=/dev/sdc of=/dev/null bs=1K count=1 iflag=direct; done
2.kill -SIGSTOP `pidof iscsid`
3.iptables -A OUTPUT -p tcp --dport 3260 -j DROP

I think it is cause by a status conflict between the waitting for lost IO
request and iscsi device remove procedure. Any new thoughs?

-- 
You are receiving this mail because:
You are the assignee for the bug.