abort_task_set / abort_cmd problems

abort_task_set / abort_cmd problems

[Scott Friedman]

Hi

I am running tgtd-1.0.54 currently (centos 6.7,
2.6.32-573.7.1.el6.x86_64) and under heavy load to a single target and
LUN my log fills up with:

Dec  8 08:09:44 library3 tgtd: abort_task_set(1348) found 0 0
Dec  8 08:09:44 library3 tgtd: abort_cmd(1324) found 3c7e0700 6
Dec  8 08:09:44 library3 tgtd: abort_cmd(1324) found d97c0700 6
Dec  8 08:09:44 library3 tgtd: abort_cmd(1324) found c0820700 6
<snip>
Dec  8 08:09:44 library3 tgtd: abort_cmd(1324) found 77800700 6
Dec  8 08:09:44 library3 tgtd: abort_cmd(1324) found e17f0700 6
Dec  8 08:10:04 library3 tgtd: conn_close(92) already closed 0x2495ab0 3

This sequence will happen every minute or so under writes from the
initiator (a windows 2008 r2 server). The backing store is a ZFS ZVOL
and it does not report any issues.

As time goes on the the system load continues to increase (I have seem
it in the low teens on a dual core VM).

The result of this is that there ends up being a lot of (I guess)
stale sessions - which are unremovable. The only we have found to
truely get rid of them is to reboot.

    I_T nexus information:
        I_T nexus: 2
            Initiator: iqn.blahblah alias: none
            Connection: 1
                IP Address: 1.2.3.4
        I_T nexus: 4
            Initiator: iqn.blahblah alias: none
            Connection: 1
                IP Address: 1.2.3.4
        I_T nexus: 6
            Initiator: iqn.blahblah alias: none
            Connection: 1
                IP Address: 1.2.3.4
        I_T nexus: 7
            Initiator: iqn.blahblah alias: none
            Connection: 1
                IP Address: 1.2.3.4

The fact that some of the sid seem to be gone makes me think that
sometimes the sessions get removed properly and some do not. The
initiator does complain about timeouts that correspond to the logging
above.

I figured it is time to run tgtd in debug mode but do not know what
level to use or what to look for.

Any pointers would be very appreciated as we cannot keep this system
up for more than a couple hours.

Thanks,
Scott