Re: 2.6.24-rc3-mm1: I/O error, system hangs

[Gabriel C <nix.or.die@googlemail.com>]

Gabriel C wrote:
> James Bottomley wrote:
>> On Sat, 2007-11-24 at 18:54 +0100, Gabriel C wrote:
>>> James Bottomley wrote:
>>>> On Sat, 2007-11-24 at 13:57 +0100, Laurent Riffard wrote:
>>>>> Le 24.11.2007 07:42, James Bottomley a écrit :
>>>>>> On Fri, 2007-11-23 at 18:52 +0100, Laurent Riffard wrote:
>>>>>>> Le 23.11.2007 12:38, Hannes Reinecke a écrit :
>>>>>>>> Hannes Reinecke wrote:
>>>>>>>>> Laurent Riffard wrote:
>>>>>>>>>> Le 21.11.2007 23:41, Andrew Morton a écrit :
>>>>>>>>>>> On Wed, 21 Nov 2007 22:45:22 +0100
>>>>>>>>>>> Laurent Riffard <laurent.riffard@free.fr> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Le 21.11.2007 05:45, Andrew Morton a écrit :
>>>>>>>>>>>>> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.24-rc3/2.6.24-rc3-mm1/
>>>>>>>>>>>> Hello, 
>>>>>>>>>>>>
>>>>>>>>>>>> My system hangs shortly after I logged in Gnome desktop. SysRq-W shows
>>>>>>>>>>>> that a bunch of task are blocked in "D" state, they seem to wait for
>>>>>>>>>>>> some I/O completion. I can try to hand-copy some data if requested.
>>>>>>>>>>>>
>>>>>>>>>>>> I found these messages in dmesg:
>>>>>>>>>>>>
>>>>>>>>>>>> ~$ grep -C2 end_request dmesg-2.6.24-rc3-mm1 
>>>>>>>>>>>> EXT3-fs: mounted filesystem with ordered data mode.
>>>>>>>>>>>> sd 0:0:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
>>>>>>>>>>>> end_request: I/O error, dev sda, sector 16460
>>>>>>>>>>>> ReiserFS: sda7: found reiserfs format "3.6" with standard journal
>>>>>>>>>>>> ReiserFS: sda7: using ordered data mode
>>>>>>>>>>>> --
>>>>>>>>>>>> ReiserFS: sda7: Using r5 hash to sort names
>>>>>>>>>>>> sd 0:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
>>>>>>>>>>>> end_request: I/O error, dev sdb, sector 19632
>>>>>>>>>>>> sd 0:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
>>>>>>>>>>>> end_request: I/O error, dev sdb, sector 40037363
>>>>>>>>>>>> Adding 1048568k swap on /dev/mapper/vglinux1-lvswap.  Priority:-1 extents:1 across:1048568k
>>>>>>>>>>>> lp0: using parport0 (interrupt-driven).
>>>>>>>>>>>>
>>>>>>>>>>>> These errors occur *only* with 2.6.24-rc3-mm1, they are 100% reproducible.
>>>>>>>>>>>> 2.6.24-rc3 and 2.6.24-rc2-mm1 are fine.
>>>>>>>>>>>>
>>>>>>>>>>>> Maybe something is broken in pata_via driver ?
>>>>>>>>>>>>
>>>>>>>>>>> Could be - libata-reimplement-ata_acpi_cbl_80wire-using-ata_acpi_gtm_xfermask.patch
>>>>>>>>>>> and pata_amd-pata_via-de-couple-programming-of-pio-mwdma-and-udma-timings.patch
>>>>>>>>>>> touch pata_via.c.
>>>>>>>>>> None of the above...
>>>>>>>>>>
>>>>>>>>>> I did a bisection, it spotted git-scsi-misc.patch. 
>>>>>>>>>> I just run 2.6.24-rc3-mm1 + revert-git-scsi-misc.patch, and it works fine.
>>>>>>>>>>
>>>>>>>>>> I guess commit 8655a546c83fc43f0a73416bbd126d02de7ad6c0 "[SCSI] Do not 
>>>>>>>>>> requeue requests if REQ_FAILFAST is set" is the real culprit. The other 
>>>>>>>>>> commits are touching documentation or drivers I don't use. I'll try 
>>>>>>>>>> to revert only this one this evening.
>>>>>>> I can confirm : reverting commit 8655a546c83fc43f0a73416bbd126d02de7ad6c0 
>>>>>>> does fix the problem.
>>>>>>>
>>>>>>>>> Hmm. Weird. I'll have a look into it. Apparently I'll be returning an error where
>>>>>>>>> I shouldn't. Checking ...
>>>>>>>>>
>>>>>>>> Ok, found it. We are blocking even special commands (ie requests with PREEMPT not set)
>>>>>>>> when FAILFAST is set. Which is clearly wrong. The attached patch fixes this.
>>>>>>> Sorry, it's not enough. 2.6.24-rc3-mm1 + your patch still hangs with I/O errors.
>>>>>> I think the problem is the way we treat BLOCKED and QUIESCED (the latter
>>>>>> is the state that the domain validation uses and which we cannot kill
>>>>>> fastfail on).  It's definitely wrong to kill fastfail requests when the
>>>>>> state is QUIESCE.
>>>>>>
>>>>>> This patch (which is applied on top of Hannes original) separates the
>>>>>> BLOCK and QUIESCE states correctly ... does this fix the problem?
>>>>> No, it doesn't help... (2.6.24-rc3-mm1 + your patch still has problems)
>>>> OK, could you post dmesgs again, please.  I actually tested this with an
>>>> aic79xx card, and for me it does cause Domain Validation to succeed
>>>> again.
>>>>
>>> Are the patches indeed to fix that problem as well ? 
>>>
>>> http://lkml.org/lkml/2007/11/23/5
>> That dmesg is from an unknown SCSI card exhibiting Domain Validation
>> problems, so it's a reasonable probability, yes ... but you'll need the
>> additional hack I just did to prevent further intermittent failures.
> 
> My controller is:
> 
> 03:0e.0 SCSI storage controller [0100]: Adaptec AIC-7892P U160/m [9005:008f] (rev 02)
> 
> I'll try the patches in a bit.

With your patches my problem(s) are solved. Domain Validation works again.

...

[   32.179521] scsi 0:0:0:0: Direct-Access     SEAGATE  ST318406LW       0109 PQ: 0 ANSI: 3
[   32.179540] scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
[   32.179554]  target0:0:0: Beginning Domain Validation
[   32.188553]  target0:0:0: wide asynchronous
[   32.195302]  target0:0:0: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 63)
[   32.206510]  target0:0:0: Ending Domain Validation
[   32.211699] scsi 0:0:1:0: Direct-Access     FUJITSU  MAH3182MP        0114 PQ: 0 ANSI: 4
[   32.211707] scsi0:A:1:0: Tagged Queuing enabled.  Depth 32
[   32.211717]  target0:0:1: Beginning Domain Validation
[   32.213980]  target0:0:1: wide asynchronous
[   32.215682]  target0:0:1: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   32.220205]  target0:0:1: Ending Domain Validation

...

 Thx James :)


Gabriel