How to online remove an error scsi disk from the system?

How to online remove an error scsi disk from the system?

[Tao Ma]

Hi All,
	In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will
hang forever and we will get something in /var/log/messages like below.
It seems to me that the io sent to the scsi layer is never returned back
with -EIO which is a little bit surprised for me(It should be a timeout
somewhere, right?). We have tried echo "offline" >
/sys/block/sdl/device/state, but it doesn't work. So is there any way
for us to let the scsi device returns all the io requests back with EIO
so that all the end_io can be called accordingly? Am I missing something
here?

Thanks,
Tao


sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580)
sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00
target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11)
target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)
INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdl1-8   D 0000000000000000     0  4629      2 0x00000000
 ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000
 ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080
 ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000
Call Trace:
 [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff814a8a53>] io_schedule+0x73/0xc0
 [<ffffffff811036a8>] sync_page+0x38/0x50
 [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
 [<ffffffff81103670>] ? sync_page+0x0/0x50
 [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
 [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
 [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
 [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
 [<ffffffff811172f0>] ? __writepage+0x0/0x40
 [<ffffffff81118de4>] generic_writepages+0x24/0x30
 [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]
 [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
 [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
 [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81089166>] kthread+0x96/0xa0
 [<ffffffff8100c08a>] child_rip+0xa/0x20
 [<ffffffff810890d0>] ? kthread+0x0/0xa0
 [<ffffffff8100c080>] ? child_rip+0x0/0x20

Re: How to online remove an error scsi disk from the system?

[Bart Van Assche]

On 02/01/13 07:13, Tao Ma wrote:
> 	In our product system, we have several sata disks attached to one
> machine. So when one of the disk fails, the jbd2(yes, we use ext4) will
> hang forever and we will get something in /var/log/messages like below.
> It seems to me that the io sent to the scsi layer is never returned back
> with -EIO which is a little bit surprised for me(It should be a timeout
> somewhere, right?). We have tried echo "offline" >
> /sys/block/sdl/device/state, but it doesn't work. So is there any way
> for us to let the scsi device returns all the io requests back with EIO
> so that all the end_io can be called accordingly? Am I missing something
> here?

Please note that I'm not familiar with SAS. But I found this in 
drivers/scsi/scsi_proc.c:

  * proc_scsi_write - handle writes to /proc/scsi/scsi
  * @file: not used
  * @buf: buffer to write
  * @length: length of buf, at most PAGE_SIZE
  * @ppos: not used
  *
  * Description: this provides a legacy mechanism to add or remove
  * devices by Host, Channel, ID, and Lun.  To use,
  * "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
  * "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
  * "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

Bart.

RE: How to online remove an error scsi disk from the system?

[Jack Wang]

Hi All,
	In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang
forever and we will get something in /var/log/messages like below.
It seems to me that the io sent to the scsi layer is never returned back
with -EIO which is a little bit surprised for me(It should be a timeout
somewhere, right?). We have tried echo "offline" >
/sys/block/sdl/device/state, but it doesn't work. So is there any way for us
to let the scsi device returns all the io requests back with EIO so that all
the end_io can be called accordingly? Am I missing something here?

Thanks,
Tao
[Jack Wang] 
Hi Tao,

Have you tried:
 echo 1 > /sys/block/sdv/device/delete
 echo "- - -" > /sys/class/scsi_host/host

another way is :
find out which phy the disk attached to and:
echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset

Jack

Re: How to online remove an error scsi disk from the system?

[Tao Ma]

On 02/01/2013 03:54 PM, Bart Van Assche wrote:
> On 02/01/13 07:13, Tao Ma wrote:
>>     In our product system, we have several sata disks attached to one
>> machine. So when one of the disk fails, the jbd2(yes, we use ext4) will
>> hang forever and we will get something in /var/log/messages like below.
>> It seems to me that the io sent to the scsi layer is never returned back
>> with -EIO which is a little bit surprised for me(It should be a timeout
>> somewhere, right?). We have tried echo "offline" >
>> /sys/block/sdl/device/state, but it doesn't work. So is there any way
>> for us to let the scsi device returns all the io requests back with EIO
>> so that all the end_io can be called accordingly? Am I missing something
>> here?
> 
> Please note that I'm not familiar with SAS. But I found this in
> drivers/scsi/scsi_proc.c:
> 
>  * proc_scsi_write - handle writes to /proc/scsi/scsi
>  * @file: not used
>  * @buf: buffer to write
>  * @length: length of buf, at most PAGE_SIZE
>  * @ppos: not used
>  *
>  * Description: this provides a legacy mechanism to add or remove
>  * devices by Host, Channel, ID, and Lun.  To use,
>  * "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
>  * "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
>  * "0 1 2 3" replaced by the Host, Channel, Id, and Lun.
Sorry, it doesn't work since it will also send some IOs to the scsi. And
it hangs...

bash          D 0000000000000000     0 57479  57477 0x00000000
 ffff8817fee2dba0 0000000000000086 0000000000000000 0000000000000002
 ffffffff817c4ed5 0000000000015f40 ffff88180c7e45f8 ffff88180c7e4040
 ffffffff81a2d020 ffff88180c7e45f8 000000010fa4af09 0000000000000004
Call Trace:
 [<ffffffff8123eecf>] ? string+0x3f/0xd0
 [<ffffffff8123fdc2>] ? vsnprintf+0x242/0x580
 [<ffffffff811a0b14>] ? fsnotify_clear_marks_by_inode+0x34/0xf0
 [<ffffffff811d33c0>] ? sysfs_delete_inode+0x0/0x60
 [<ffffffff814aa5c5>] rwsem_down_failed_common+0x95/0x1c0
 [<ffffffff814aa746>] rwsem_down_read_failed+0x26/0x30
 [<ffffffff81241814>] call_rwsem_down_read_failed+0x14/0x30
 [<ffffffff812385a0>] ? kobject_release+0x0/0x1f0
 [<ffffffff814a9cd4>] ? down_read+0x24/0x30
 [<ffffffff81167794>] get_super+0x74/0xc0
 [<ffffffff8119aa9e>] fsync_bdev+0x1e/0x60
 [<ffffffff812253ce>] invalidate_partition+0x2e/0x60
 [<ffffffff811d0bfe>] del_gendisk+0x3e/0x130
 [<ffffffff813070da>] ? device_del+0x16a/0x1a0
 [<ffffffff8132f437>] sd_remove+0x67/0xb0
 [<ffffffff8130adcf>] __device_release_driver+0x6f/0xe0
 [<ffffffff8130ae6d>] device_release_driver+0x2d/0x40
 [<ffffffff8130a723>] bus_remove_device+0x83/0xe0
 [<ffffffff8130709f>] device_del+0x12f/0x1a0
 [<ffffffff8132a7f5>] __scsi_remove_device+0xa5/0xb0
 [<ffffffff8132a830>] scsi_remove_device+0x30/0x50
 [<ffffffff8132cc3f>] proc_scsi_write+0x23f/0x280
 [<ffffffff81182869>] ? mntput_no_expire+0x39/0xd0
 [<ffffffff811c482f>] proc_reg_write+0x7f/0xc0
 [<ffffffff81165c6c>] vfs_write+0xcc/0x1a0
 [<ffffffff81165e25>] sys_write+0x55/0x90
 [<ffffffff8100b032>] system_call_fastpath+0x16/0x1b


Thanks,
Tao
> 
> Bart.
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: How to online remove an error scsi disk from the system?

[Tao Ma]

On 02/01/2013 04:50 PM, Jack Wang wrote:
> Hi All,
> 	In our product system, we have several sata disks attached to one
> machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang
> forever and we will get something in /var/log/messages like below.
> It seems to me that the io sent to the scsi layer is never returned back
> with -EIO which is a little bit surprised for me(It should be a timeout
> somewhere, right?). We have tried echo "offline" >
> /sys/block/sdl/device/state, but it doesn't work. So is there any way for us
> to let the scsi device returns all the io requests back with EIO so that all
> the end_io can be called accordingly? Am I missing something here?
> 
> Thanks,
> Tao
> [Jack Wang] 
> Hi Tao,
> 
> Have you tried:
>  echo 1 > /sys/block/sdv/device/delete
It will do some IO first so it will hang doing IO.
>  echo "- - -" > /sys/class/scsi_host/host
What do you mean for this line?
> 
> another way is :
> find out which phy the disk attached to and:
> echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset
sorry, I have done it, but there is no response.

Thanks,
Tao
> 
> Jack
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

RE: How to online remove an error scsi disk from the system?

[Jack Wang]

On 02/01/2013 04:50 PM, Jack Wang wrote:
> Hi All,
> 	In our product system, we have several sata disks attached to one 
> machine. So when one of the disk fails, the jbd2(yes, we use ext4) 
> will hang forever and we will get something in /var/log/messages like
below.
> It seems to me that the io sent to the scsi layer is never returned 
> back with -EIO which is a little bit surprised for me(It should be a 
> timeout somewhere, right?). We have tried echo "offline" > 
> /sys/block/sdl/device/state, but it doesn't work. So is there any way 
> for us to let the scsi device returns all the io requests back with 
> EIO so that all the end_io can be called accordingly? Am I missing
something here?
> 
> Thanks,
> Tao
> [Jack Wang]
> Hi Tao,
> 
> Have you tried:
>  echo 1 > /sys/block/sdv/device/delete
It will do some IO first so it will hang doing IO.
>  echo "- - -" > /sys/class/scsi_host/host
What do you mean for this line?

[Jack Wang] Sorry I mean to let the driver rescan to get the disk back.
The line should be :
 echo "- - -" > /sys/class/scsi_host/hostx/scan.

Per above delete does not work , so no need to run this.
> 
> another way is :
> find out which phy the disk attached to and:
> echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset
sorry, I have done it, but there is no response.

[Jack Wang] 
What about
echo 1 > /sys/class/sas_phy/phy-x:x:x/hard_reset

?
Thanks,
Tao
> 
> Jack
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to majordomo@vger.kernel.org 
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the
body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: How to online remove an error scsi disk from the system?

[Tao Ma]

On 02/01/2013 05:24 PM, Jack Wang wrote:
> 
> On 02/01/2013 04:50 PM, Jack Wang wrote:
>> Hi All,
>> 	In our product system, we have several sata disks attached to one 
>> machine. So when one of the disk fails, the jbd2(yes, we use ext4) 
>> will hang forever and we will get something in /var/log/messages like
> below.
>> It seems to me that the io sent to the scsi layer is never returned 
>> back with -EIO which is a little bit surprised for me(It should be a 
>> timeout somewhere, right?). We have tried echo "offline" > 
>> /sys/block/sdl/device/state, but it doesn't work. So is there any way 
>> for us to let the scsi device returns all the io requests back with 
>> EIO so that all the end_io can be called accordingly? Am I missing
> something here?
>>
>> Thanks,
>> Tao
>> [Jack Wang]
>> Hi Tao,
>>
>> Have you tried:
>>  echo 1 > /sys/block/sdv/device/delete
> It will do some IO first so it will hang doing IO.
>>  echo "- - -" > /sys/class/scsi_host/host
> What do you mean for this line?
> 
> [Jack Wang] Sorry I mean to let the driver rescan to get the disk back.
> The line should be :
>  echo "- - -" > /sys/class/scsi_host/hostx/scan.
> 
> Per above delete does not work , so no need to run this.
>>
>> another way is :
>> find out which phy the disk attached to and:
>> echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset
> sorry, I have done it, but there is no response.
> 
> [Jack Wang] 
> What about
> echo 1 > /sys/class/sas_phy/phy-x:x:x/hard_reset
sorry, no response either.

Thanks,
Tao
> 
> ?
> Thanks,
> Tao
>>
>> Jack
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: How to online remove an error scsi disk from the system?

[Bryn M. Reeves]

On 02/01/2013 07:54 AM, Bart Van Assche wrote:
>   * proc_scsi_write - handle writes to /proc/scsi/scsi
>   * @file: not used
>   * @buf: buffer to write
>   * @length: length of buf, at most PAGE_SIZE
>   * @ppos: not used
>   *
>   * Description: this provides a legacy mechanism to add or remove
>   * devices by Host, Channel, ID, and Lun.  To use,
>   * "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
>   * "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
>   * "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

The proc interface is deprecated; this can all be done via sysfs today, 
e.g.:

echo 1 > /sys/block/sdc/device/delete

Is equivalent to issuing scsi remove-single-device to proc.

Regards,
Bryn.

Re: How to online remove an error scsi disk from the system?

[Tao Ma]

On 02/01/2013 05:52 PM, Bryn M. Reeves wrote:
> On 02/01/2013 07:54 AM, Bart Van Assche wrote:
>>   * proc_scsi_write - handle writes to /proc/scsi/scsi
>>   * @file: not used
>>   * @buf: buffer to write
>>   * @length: length of buf, at most PAGE_SIZE
>>   * @ppos: not used
>>   *
>>   * Description: this provides a legacy mechanism to add or remove
>>   * devices by Host, Channel, ID, and Lun.  To use,
>>   * "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
>>   * "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
>>   * "0 1 2 3" replaced by the Host, Channel, Id, and Lun.
> 
> The proc interface is deprecated; this can all be done via sysfs today,
> e.g.:
> 
> echo 1 > /sys/block/sdc/device/delete
> 
> Is equivalent to issuing scsi remove-single-device to proc.
yes, but the result is the same. It will do some IO first which will
cause this command hang.

Thanks,
Tao
> 
> Regards,
> Bryn.
> 
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: How to online remove an error scsi disk from the system?

[Bryn M. Reeves]

On 02/01/2013 09:59 AM, Tao Ma wrote:
> yes, but the result is the same. It will do some IO first which will
> cause this command hang.

You seem to have a problem with either the device/adapter or in the 
driver. The backtrace you posted shows that jbd2 (ext4) is still waiting 
on IO that's been submitted to an mpt2sas or mpt3sas adapter (I only 
know that because I recognise their log messages - you should try to 
include relevant details like this when seeking assistance).

The adapter/driver hasn't completed the IO and it looks like the SCSI 
layer is trying to abort it. Depending on the state of the driver and 
hardware your only option might be to reboot (or physically hot remove 
the device if your hardware allows it).

You don't mention the versions of the kernel and driver you're using - 
if the system is in production I would suggest contacting who ever 
normally provides support for the kernel and distribution that you are 
running.

Regards,
Bryn.

Re: How to online remove an error scsi disk from the system?

[Tao Ma]

On 02/01/2013 06:07 PM, Bryn M. Reeves wrote:
> On 02/01/2013 09:59 AM, Tao Ma wrote:
>> yes, but the result is the same. It will do some IO first which will
>> cause this command hang.
> 
> You seem to have a problem with either the device/adapter or in the
> driver. The backtrace you posted shows that jbd2 (ext4) is still waiting
> on IO that's been submitted to an mpt2sas or mpt3sas adapter (I only
> know that because I recognise their log messages - you should try to
> include relevant details like this when seeking assistance).
This should be  a mpt2sas adapter
#lsmod|grep mpt
mptctl                 96789  0
mptbase                97052  1 mptctl
mpt2sas               164962  18
scsi_transport_sas     35232  3 isci,libsas,mpt2sas
raid_class              4746  1 mpt2sas

The system has 12 sata disks. What else do you need? I am willing to
provide any details you want.

> 
> The adapter/driver hasn't completed the IO and it looks like the SCSI
> layer is trying to abort it. Depending on the state of the driver and
> hardware your only option might be to reboot (or physically hot remove
> the device if your hardware allows it).
OK, so let me describe the situation here. This is one of our storage
system. So 12 2TB sata disk in one box, normally when one disk fails, we
just want to remove it from the system by *software*, and then continue
to use the 11 disks left. We have found that sometimes an unsuccessful
umount or some actions against this disk can lead to some bad
situation(Say some very high load because many processes are 'D'ed). So
ideally if we can remove this device successfully, all the ios to this
disk will fail and there will be no 'D' processes and the loadavg will
also be low.
> 
> You don't mention the versions of the kernel and driver you're using -
> if the system is in production I would suggest contacting who ever
> normally provides support for the kernel and distribution that you are
> running.
We use CentOS6.2 and the kernel version is 2.6.32-220.23.1.

Thanks,
Tao

Re: How to online remove an error scsi disk from the system?

[Bryn M. Reeves]

On 02/01/2013 11:13 AM, Tao Ma wrote:
>> You don't mention the versions of the kernel and driver you're using -
>> if the system is in production I would suggest contacting who ever
>> normally provides support for the kernel and distribution that you are
>> running.
> We use CentOS6.2 and the kernel version is 2.6.32-220.23.1.

This is ancient, even by CentOS or RHEL standards. There are thousands 
of patches in more recent kernels (either at kernel.org or in the 
updates in CentOS repositories).

Nobody on linux-kernel or the other lists you copied is going to want to 
investigate problems on such an old kernel - you'll need to either 
reproduce with something current or seek assistance from the CentOS 
community (who will probably tell you to update your kernel first anyway).

Regards,
Bryn.

Re: How to online remove an error scsi disk from the system?

[Hillf Danton]

On Fri, Feb 1, 2013 at 2:13 PM, Tao Ma <tm@tao.ma> wrote:
> Hi All,
>         In our product system, we have several sata disks attached to one
> machine. So when one of the disk fails, the jbd2(yes, we use ext4) will
> hang forever and we will get something in /var/log/messages like below.
> It seems to me that the io sent to the scsi layer is never returned back
> with -EIO which is a little bit surprised for me(It should be a timeout
> somewhere, right?). We have tried echo "offline" >
> /sys/block/sdl/device/state, but it doesn't work. So is there any way
> for us to let the scsi device returns all the io requests back with EIO
> so that all the end_io can be called accordingly? Am I missing something
> here?
>
> Thanks,
> Tao
>
>
> sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580)
> sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00
> target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11)
> target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)
> INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> jbd2/sdl1-8   D 0000000000000000     0  4629      2 0x00000000
>  ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000
>  ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080
>  ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000
> Call Trace:
>  [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
>  [<ffffffff810122b6>] ? read_tsc+0x16/0x40
>  [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
>  [<ffffffff810122b6>] ? read_tsc+0x16/0x40
>  [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
>  [<ffffffff814a8a53>] io_schedule+0x73/0xc0
>  [<ffffffff811036a8>] sync_page+0x38/0x50
>  [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
>  [<ffffffff81103670>] ? sync_page+0x0/0x50
>  [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
>  [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
>  [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
>  [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
>  [<ffffffff811172f0>] ? __writepage+0x0/0x40
>  [<ffffffff81118de4>] generic_writepages+0x24/0x30
>  [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]
>  [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
>  [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
>  [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
>  [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
>  [<ffffffff81089166>] kthread+0x96/0xa0
>  [<ffffffff8100c08a>] child_rip+0xa/0x20
>  [<ffffffff810890d0>] ? kthread+0x0/0xa0
>  [<ffffffff8100c080>] ? child_rip+0x0/0x20
>
Can you try upstream?

Re: How to online remove an error scsi disk from the system?

[taco]

On Fri, Feb 01, 2013 at 02:13:16PM +0800, Tao Ma wrote:
> Hi All,
> 	In our product system, we have several sata disks attached to one
> machine. So when one of the disk fails, the jbd2(yes, we use ext4) will
> hang forever and we will get something in /var/log/messages like below.
> It seems to me that the io sent to the scsi layer is never returned back
> with -EIO which is a little bit surprised for me(It should be a timeout
> somewhere, right?). We have tried echo "offline" >
> /sys/block/sdl/device/state, but it doesn't work. So is there any way
> for us to let the scsi device returns all the io requests back with EIO
> so that all the end_io can be called accordingly? Am I missing something
> here?
> 
> Thanks,
> Tao
> 
> 
> sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580)
It seems that IO timeout cause HBA's driver to abort scmd,
the aborted IO came back with scmd->result = DID_RESET << 16;
with this result code the Middle layer of scsi will retry this IO.
IO timeout again due to Bad disk so, this IO loop forever and
never come back.

might it is a bug of mpt2sas driver.

> sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00
> target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11)
> target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)
> INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> jbd2/sdl1-8   D 0000000000000000     0  4629      2 0x00000000
>  ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000
>  ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080
>  ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000
> Call Trace:
>  [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
>  [<ffffffff810122b6>] ? read_tsc+0x16/0x40
>  [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
>  [<ffffffff810122b6>] ? read_tsc+0x16/0x40
>  [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
>  [<ffffffff814a8a53>] io_schedule+0x73/0xc0
>  [<ffffffff811036a8>] sync_page+0x38/0x50
>  [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
>  [<ffffffff81103670>] ? sync_page+0x0/0x50
>  [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
>  [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
>  [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
>  [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
>  [<ffffffff811172f0>] ? __writepage+0x0/0x40
>  [<ffffffff81118de4>] generic_writepages+0x24/0x30
>  [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]
>  [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
>  [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
>  [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
>  [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
>  [<ffffffff81089166>] kthread+0x96/0xa0
>  [<ffffffff8100c08a>] child_rip+0xa/0x20
>  [<ffffffff810890d0>] ? kthread+0x0/0xa0
>  [<ffffffff8100c080>] ? child_rip+0x0/0x20
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html