Tagged overlapped commands

Tagged overlapped commands

[Andrew Brooks]

Hi

I'm having trouble writing to an LTO-5 tape drive connected via Adaptec 1405
Writing aborts with this in syslog:

st0: Sense Key : Aborted Command [current]
st0: Add. Sense: Tagged overlapped commands (task tag 0)

Both the drive and adapter have the latest firmware.
I've tried ubuntu 12.04 (3.2.0-70) and the latest fedora 20.1.

Does anyone have any ideas on how to troubleshoot and fix this?

Thanks

Andrew

Re: Tagged overlapped commands

[Andrew Brooks]

On 1 December 2014 at 12:24, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
>
> I'm having trouble writing to an LTO-5 tape drive connected via Adaptec 1405
> Writing aborts with this in syslog:
>
> st0: Sense Key : Aborted Command [current]
> st0: Add. Sense: Tagged overlapped commands (task tag 0)

If I enable debugging I get the following, does it help narrow down the problem?

14:02:25 kernel: [ 1372.991912] mv_sas.c 2067:phy 1 ctrl sts=0x00089800.
14:02:25 kernel: [ 1372.991914] mv_sas.c 2069:phy 1 irq sts = 0x01001001
14:02:25 kernel: [ 1372.995158] mv_sas.c 1957:Release slot [0] tag[0],
task [ffff88021e2161c0]:
14:02:25 kernel: [ 1373.488140] st0: Error 10000 (driver bt 0x0, host bt 0x1).
14:02:25 kernel: [ 1373.488432] mv_sas.c 753:SAS port 0 does not attachdevice.
14:02:25 kernel: [ 1373.490299] st0: Error 10000 (driver bt 0x0, host bt 0x1).
14:02:25 kernel: [ 1373.490302] st0: Error on write filemark.
14:02:25 kernel: [ 1373.506689] mv_sas.c 753:SAS port 0 does not attachdevice.
14:02:25 kernel: [ 1373.506707] st0: Error 10000 (driver bt 0x0, host bt 0x1).
14:02:27 kernel: [ 1375.392013] mv_sas.c 2005:phy1 Removed Device
14:02:27 kernel: [ 1375.395238] mv_sas.c 1353:found dev[0:1] is gone.
14:02:30 kernel: [ 1379.172027] mv_sas.c 2069:phy 1 irq sts = 0x00000085
14:02:30 kernel: [ 1379.172033] mv_sas.c 2118:notify plug in on phy[1]
14:02:30 kernel: [ 1379.176012] mv_sas.c 1181:phy 1 attach dev info is 80001
14:02:30 kernel: [ 1379.176012] mv_sas.c 1183:phy 1 attach sas addr is
7c325301a0100150
14:02:30 kernel: [ 1379.176012] mv_sas.c 277:phy 1 byte dmaded.
14:02:30 kernel: [ 1379.261430] sas: phy-2:1 added to port-2:0,
phy_mask:0x2 (500110a00153327c)
14:02:30 kernel: [ 1379.261433] mv_sas.c 1218:set wide port phy map 2
14:02:30 kernel: [ 1379.291229] sas: DOING DISCOVERY on port 0, pid:17
14:02:30 kernel: [ 1379.292725] scsi 2:0:1:0: Sequential-Access HP
Ultrium 5-SCSI   Z64D PQ: 0 ANSI: 6
14:02:30 kernel: [ 1379.296458] st 2:0:1:0: Attached scsi tape st0
14:02:30 kernel: [ 1379.296462] st 2:0:1:0: st0: try direct i/o: yes
(alignment 4 B)
14:02:30 kernel: [ 1379.296531] st 2:0:1:0: Attached scsi generic sg2 type 1
14:02:30 kernel: [ 1379.297314] sas: DONE DISCOVERY on port 0, pid:17, result:0
14:02:35 kernel: [ 1384.176006] mv_sas.c 2053:Get signature time out,
reset phy 1
14:02:35 kernel: [ 1384.176020] mv_sas.c 2067:phy 1 ctrl sts=0x00089800.
14:02:35 kernel: [ 1384.176024] mv_sas.c 2069:phy 1 irq sts = 0x00001001
14:02:36 kernel: [ 1384.672106] mv_sas.c 2069:phy 1 irq sts = 0x0100109F
14:02:36 kernel: [ 1384.676094] mv_sas.c 2118:notify plug in on phy[1]
14:02:36 kernel: [ 1384.676094] mv_sas.c 2143:plugin interrupt but phy1 is gone
14:02:36 kernel: [ 1385.168221] mv_sas.c 753:SAS port 0 does not attachdevice.
14:02:37 kernel: [ 1386.172012] mv_sas.c 753:SAS port 0 does not attachdevice.
14:02:38 kernel: [ 1386.496002] mv_sas.c 1181:phy 1 attach dev info is 80001
14:02:38 kernel: [ 1386.496002] mv_sas.c 1183:phy 1 attach sas addr is
7c325301a0100150
14:02:38 kernel: [ 1386.496002] mv_sas.c 277:phy 1 byte dmaded.
14:02:38 kernel: [ 1386.551563] mv_sas.c 1218:set wide port phy map 2
14:02:38 kernel: [ 1386.561489] sas: sas_form_port: phy1 belongs to
port0 already(1)!
14:02:38 kernel: [ 1386.551563] mv_sas.c 2011:phy1 Attached Device

Re: Tagged overlapped commands

[James Bottomley]

On Mon, 2014-12-01 at 12:24 +0000, Andrew Brooks wrote:
> Hi
> 
> I'm having trouble writing to an LTO-5 tape drive connected via Adaptec 1405
> Writing aborts with this in syslog:
> 
> st0: Sense Key : Aborted Command [current]
> st0: Add. Sense: Tagged overlapped commands (task tag 0)
> 
> Both the drive and adapter have the latest firmware.
> I've tried ubuntu 12.04 (3.2.0-70) and the latest fedora 20.1.
> 
> Does anyone have any ideas on how to troubleshoot and fix this?

The error message likely means that the tape device had more than one
command outstanding at once (i.e. one command was sent while the current
command was still pending).  It could also mean that a tag got re-used
(same tag issued while command with that tag was outstanding) but for a
tape I think the former is more likely.  Is the queue depth set to one
for the device?

James

Re: Tagged overlapped commands

[James Bottomley]

Adding linux-scsi back to cc.  Please don't drop because others may be
interested and it allows search engines to find the answer.

On Fri, 2014-12-05 at 13:09 +0000, Andrew Brooks wrote:
> On 4 December 2014 at 16:17, James Bottomley <
> James.Bottomley@hansenpartnership.com> wrote:
> 
> > The error message likely means that the tape device had more than one
> > command outstanding at once (i.e. one command was sent while the current
> > command was still pending).  It could also mean that a tag got re-used
> > (same tag issued while command with that tag was outstanding) but for a
> > tape I think the former is more likely.  Is the queue depth set to one
> > for the device?
> >
> 
> Thanks for your reply, How can I configure the queue depth?

it should be under

/sys/class/scsi_device/<dev>/device/queue_depth

you can (as root) echo a value in there to change the depth.

James

Re: Tagged overlapped commands

[Andrew Brooks]

On 8 December 2014 at 07:02, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> The error message likely means that the tape device had more than one
> command outstanding at once (i.e. one command was sent while the current
> command was still pending).  It could also mean that a tag got re-used
> (same tag issued while command with that tag was outstanding) but for a
> tape I think the former is more likely.  Is the queue depth set to one
> for the device?
>
> /sys/class/scsi_device/<dev>/device/queue_depth
> you can (as root) echo a value in there to change the depth.

Sadly setting the queue depth to 1 didn't prevent the error message :-(
All other ideas and suggestions for debugging gratefully received!

st0: Sense Key : Aborted Command [current]
st0: Add. Sense: Tagged overlapped commands (task tag 0)

RE: Tagged overlapped commands

[Elliott, Robert (Server Storage)]

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
> owner@vger.kernel.org] On Behalf Of Andrew Brooks
> Sent: Tuesday, 16 December, 2014 4:23 AM
...
> 
> On 8 December 2014 at 07:02, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
> >
> > The error message likely means that the tape device had more than one
> > command outstanding at once (i.e. one command was sent while the current
> > command was still pending).  It could also mean that a tag got re-used
> > (same tag issued while command with that tag was outstanding) but for a
> > tape I think the former is more likely.  Is the queue depth set to one
> > for the device?
> >
...
> 
> st0: Sense Key : Aborted Command [current]
> st0: Add. Sense: Tagged overlapped commands (task tag 0)

That error could be due to:
1. send command with tag 0
2. timeout command with tag 0
3. abort command with tag 0, doesn't get through
4. send new command with tag 0

What are the serial log messages preceding those messages?

For example, if you see this print from mv_sas.c mvs_abort_task():
                if (rc == 0) {
                        mv_printk("No such tag in %s\n", __func__);
                        rc = TMF_RESP_FUNC_FAILED;
                        return rc;
                }
and this from libsas.c:
                        case TMF_RESP_FUNC_FAILED:
                                SAS_DPRINTK("%s: task 0x%p failed to abort\n",
                                                __func__, task);
                                return TASK_ABORT_FAILED;
                        }

then it didn't send the ABORT TASK at all.  If it really needed
to, that could lead to an overlap in tag 0.

BTW, in SCSI architecture, ABORT TASK is supposed to return 
a service response of FUNCTION COMPLETE rather than an error
if the tag is not present (e.g., because the command has
completed by the time the ABORT TASK was received), so the 
mvs_abort_task() handling there is questionable.

---
Rob Elliott    HP Server Storage

Re: Tagged overlapped commands

[Andrew Brooks]

On 1 December 2014 at 12:22, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
> I'm having trouble writing to an LTO-5 tape drive connected via Adaptec 1405
> Writing aborts with this in syslog:
>
> st0: Sense Key : Aborted Command [current]
> st0: Add. Sense: Tagged overlapped commands (task tag 0)
>
> Both the drive and adapter have the latest firmware.
> I've tried ubuntu 12.04 (3.2.0-70) and the latest fedora 20.1.
>
> Does anyone have any ideas on how to troubleshoot and fix this?

I've tried the latest 3.18.0 kernel and tried setting the queue_depth to 1
but it still fails.

I've tried the mvsas driver and the adaptec open-source driver but both fail.
Maybe it's a fault in the underlying scsi driver?

Please could someone offer some assistance in helping me find the cause.

Thank you

Andrew