Re: RAID 6 Failure follow up

[Andrew Dunn <andrew.g.dunn@gmail.com>]

New data now, I got this from dmesg when it went down again. Hopefully
there is some significance to you guys.

> [14269.650381] sd 10:0:3:0: rejecting I/O to offline device
> [14269.650453] sd 10:0:3:0: rejecting I/O to offline device
> [14269.650524] sd 10:0:3:0: rejecting I/O to offline device
> [14269.650595] sd 10:0:3:0: rejecting I/O to offline device
> [14269.650672] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.650675] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.650680] end_request: I/O error, dev sdh, sector 1435085631
> [14269.650749] raid5:md0: read error not correctable (sector
1435085568 on sdh1).
> [14269.650753] raid5: Disk failure on sdh1, disabling device.
> [14269.650754] raid5: Operation continuing on 7 devices.
> [14269.650886] raid5:md0: read error not correctable (sector
1435085576 on sdh1).
> [14269.650890] raid5:md0: read error not correctable (sector
1435085584 on sdh1).
> [14269.650894] raid5:md0: read error not correctable (sector
1435085592 on sdh1).
> [14269.650898] raid5:md0: read error not correctable (sector
1435085600 on sdh1).
> [14269.650902] raid5:md0: read error not correctable (sector
1435085608 on sdh1).
> [14269.650905] raid5:md0: read error not correctable (sector
1435085616 on sdh1).
> [14269.650909] raid5:md0: read error not correctable (sector
1435085624 on sdh1).
> [14269.650913] raid5:md0: read error not correctable (sector
1435085632 on sdh1).
> [14269.650917] raid5:md0: read error not correctable (sector
1435085640 on sdh1).
> [14269.650943] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.650946] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.650950] end_request: I/O error, dev sdh, sector 1435085887
> [14269.651049] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651051] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651055] end_request: I/O error, dev sdh, sector 1435086143
> [14269.651151] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651153] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651157] end_request: I/O error, dev sdh, sector 1435086399
> [14269.651253] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651255] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651259] end_request: I/O error, dev sdh, sector 1435086655
> [14269.651358] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651361] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651364] end_request: I/O error, dev sdh, sector 1435086911
> [14269.651461] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651463] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651467] end_request: I/O error, dev sdh, sector 1435087167
> [14269.651565] sd 10:0:3:0: [sdh] Unhandled error code
> [14269.651568] sd 10:0:3:0: [sdh] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
> [14269.651571] end_request: I/O error, dev sdh, sector 1435087423
> [14269.670675] end_request: I/O error, dev sdf, sector 1953519935
> [14269.670739] md: super_written gets error=-5, uptodate=0
> [14269.670743] raid5: Disk failure on sdf1, disabling device.
> [14269.670745] raid5: Operation continuing on 6 devices.
> [14269.672525] end_request: I/O error, dev sdg, sector 1953519935
> [14269.672598] md: super_written gets error=-5, uptodate=0
> [14269.672603] raid5: Disk failure on sdg1, disabling device.
> [14269.672605] raid5: Operation continuing on 5 devices.
> [14269.674402] end_request: I/O error, dev sde, sector 1953519935
> [14269.674474] md: super_written gets error=-5, uptodate=0
> [14269.674478] raid5: Disk failure on sde1, disabling device.
> [14269.674480] raid5: Operation continuing on 4 devices.
> [14269.769991] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14269.769997] Descriptor sense data with sense descriptors (in hex):
> [14269.770000]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14269.770012]         00 4f 00 c2 00 50
> [14269.770018] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14269.800245] md: md0: recovery done.
> [14269.869990] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14269.869997] Descriptor sense data with sense descriptors (in hex):
> [14269.870008]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14269.870019]         00 4f 00 c2 00 50
> [14269.870025] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14269.905144] RAID5 conf printout:
> [14269.905148]  --- rd:9 wd:4
> [14269.905152]  disk 0, o:0, dev:sde1
> [14269.905155]  disk 1, o:0, dev:sdf1
> [14269.905157]  disk 2, o:0, dev:sdg1
> [14269.905160]  disk 3, o:0, dev:sdh1
> [14269.905162]  disk 4, o:1, dev:sdi1
> [14269.905165]  disk 5, o:1, dev:sdj1
> [14269.905167]  disk 6, o:1, dev:sdk1
> [14269.905169]  disk 7, o:1, dev:sdl1
> [14269.905172]  disk 8, o:1, dev:sdm1
> [14269.941265] RAID5 conf printout:
> [14269.941269]  --- rd:9 wd:4
> [14269.941273]  disk 0, o:0, dev:sde1
> [14269.941276]  disk 1, o:0, dev:sdf1
> [14269.941278]  disk 2, o:0, dev:sdg1
> [14269.941281]  disk 3, o:0, dev:sdh1
> [14269.941283]  disk 4, o:1, dev:sdi1
> [14269.941286]  disk 5, o:1, dev:sdj1
> [14269.941289]  disk 7, o:1, dev:sdl1
> [14269.941291]  disk 8, o:1, dev:sdm1
> [14269.941300] RAID5 conf printout:
> [14269.941302]  --- rd:9 wd:4
> [14269.941304]  disk 0, o:0, dev:sde1
> [14269.941307]  disk 1, o:0, dev:sdf1
> [14269.941309]  disk 2, o:0, dev:sdg1
> [14269.941311]  disk 3, o:0, dev:sdh1
> [14269.941314]  disk 4, o:1, dev:sdi1
> [14269.941316]  disk 5, o:1, dev:sdj1
> [14269.941318]  disk 7, o:1, dev:sdl1
> [14269.941321]  disk 8, o:1, dev:sdm1
> [14269.981260] RAID5 conf printout:
> [14269.981263]  --- rd:9 wd:4
> [14269.981265]  disk 0, o:0, dev:sde1
> [14269.981268]  disk 2, o:0, dev:sdg1
> [14269.981270]  disk 3, o:0, dev:sdh1
> [14269.981273]  disk 4, o:1, dev:sdi1
> [14269.981275]  disk 5, o:1, dev:sdj1
> [14269.981277]  disk 7, o:1, dev:sdl1
> [14269.981280]  disk 8, o:1, dev:sdm1
> [14269.981284] RAID5 conf printout:
> [14269.981286]  --- rd:9 wd:4
> [14269.981289]  disk 0, o:0, dev:sde1
> [14269.981291]  disk 2, o:0, dev:sdg1
> [14269.981293]  disk 3, o:0, dev:sdh1
> [14269.981296]  disk 4, o:1, dev:sdi1
> [14269.981298]  disk 5, o:1, dev:sdj1
> [14269.981300]  disk 7, o:1, dev:sdl1
> [14269.981302]  disk 8, o:1, dev:sdm1
> [14270.003316] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14270.003324] Descriptor sense data with sense descriptors (in hex):
> [14270.003327]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.003338]         00 4f 00 c2 00 50
> [14270.003344] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14270.021260] RAID5 conf printout:
> [14270.021263]  --- rd:9 wd:4
> [14270.021266]  disk 0, o:0, dev:sde1
> [14270.021269]  disk 3, o:0, dev:sdh1
> [14270.021271]  disk 4, o:1, dev:sdi1
> [14270.021274]  disk 5, o:1, dev:sdj1
> [14270.021276]  disk 7, o:1, dev:sdl1
> [14270.021278]  disk 8, o:1, dev:sdm1
> [14270.021283] RAID5 conf printout:
> [14270.021285]  --- rd:9 wd:4
> [14270.021287]  disk 0, o:0, dev:sde1
> [14270.021289]  disk 3, o:0, dev:sdh1
> [14270.021292]  disk 4, o:1, dev:sdi1
> [14270.021294]  disk 5, o:1, dev:sdj1
> [14270.021296]  disk 7, o:1, dev:sdl1
> [14270.021298]  disk 8, o:1, dev:sdm1
> [14270.061261] RAID5 conf printout:
> [14270.061264]  --- rd:9 wd:4
> [14270.061266]  disk 0, o:0, dev:sde1
> [14270.061269]  disk 4, o:1, dev:sdi1
> [14270.061272]  disk 5, o:1, dev:sdj1
> [14270.061274]  disk 7, o:1, dev:sdl1
> [14270.061276]  disk 8, o:1, dev:sdm1
> [14270.061281] RAID5 conf printout:
> [14270.061283]  --- rd:9 wd:4
> [14270.061285]  disk 0, o:0, dev:sde1
> [14270.061287]  disk 4, o:1, dev:sdi1
> [14270.061289]  disk 5, o:1, dev:sdj1
> [14270.061292]  disk 7, o:1, dev:sdl1
> [14270.061294]  disk 8, o:1, dev:sdm1
> [14270.061647] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14270.061653] Descriptor sense data with sense descriptors (in hex):
> [14270.061656]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.061667]         00 4f 00 c2 00 50
> [14270.061672] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14270.091263] RAID5 conf printout:
> [14270.091267]  --- rd:9 wd:4
> [14270.091271]  disk 4, o:1, dev:sdi1
> [14270.091274]  disk 5, o:1, dev:sdj1
> [14270.091276]  disk 7, o:1, dev:sdl1
> [14270.091279]  disk 8, o:1, dev:sdm1
> [14270.153319] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14270.153325] Descriptor sense data with sense descriptors (in hex):
> [14270.153328]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.153340]         00 4f 00 c2 00 50
> [14270.153346] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14270.211651] sd 11:0:0:0: [sdi] Sense Key : Recovered Error
[current] [descriptor]
> [14270.211657] Descriptor sense data with sense descriptors (in hex):
> [14270.211660]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.211671]         00 4f 00 c2 00 50
> [14270.211677] sd 11:0:0:0: [sdi] Add. Sense: ATA pass through
information available
> [14270.324057] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.324065] Descriptor sense data with sense descriptors (in hex):
> [14270.324067]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.324079]         00 4f 00 c2 00 50
> [14270.324085] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.382390] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.382396] Descriptor sense data with sense descriptors (in hex):
> [14270.382399]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.382410]         00 4f 00 c2 00 50
> [14270.382416] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.474060] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.474068] Descriptor sense data with sense descriptors (in hex):
> [14270.474071]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.474083]         00 4f 00 c2 00 50
> [14270.474089] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.532394] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.532401] Descriptor sense data with sense descriptors (in hex):
> [14270.532404]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.532415]         00 4f 00 c2 00 50
> [14270.532421] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.632394] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.632402] Descriptor sense data with sense descriptors (in hex):
> [14270.632405]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.632417]         00 4f 00 c2 00 50
> [14270.632423] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.690729] sd 11:0:1:0: [sdj] Sense Key : Recovered Error
[current] [descriptor]
> [14270.690736] Descriptor sense data with sense descriptors (in hex):
> [14270.690739]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.690751]         00 4f 00 c2 00 50
> [14270.690757] sd 11:0:1:0: [sdj] Add. Sense: ATA pass through
information available
> [14270.804065] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14270.804073] Descriptor sense data with sense descriptors (in hex):
> [14270.804076]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.804088]         00 4f 00 c2 00 50
> [14270.804094] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14270.862400] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14270.862406] Descriptor sense data with sense descriptors (in hex):
> [14270.862409]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.862420]         00 4f 00 c2 00 50
> [14270.862426] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14270.954070] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14270.954079] Descriptor sense data with sense descriptors (in hex):
> [14270.954081]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14270.954093]         00 4f 00 c2 00 50
> [14270.954099] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14271.012399] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14271.012406] Descriptor sense data with sense descriptors (in hex):
> [14271.012408]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.012420]         00 4f 00 c2 00 50
> [14271.012426] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14271.104072] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14271.104080] Descriptor sense data with sense descriptors (in hex):
> [14271.104083]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.104094]         00 4f 00 c2 00 50
> [14271.104100] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14271.162400] sd 11:0:2:0: [sdk] Sense Key : Recovered Error
[current] [descriptor]
> [14271.162407] Descriptor sense data with sense descriptors (in hex):
> [14271.162410]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.162422]         00 4f 00 c2 00 50
> [14271.162428] sd 11:0:2:0: [sdk] Add. Sense: ATA pass through
information available
> [14271.278147] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.278155] Descriptor sense data with sense descriptors (in hex):
> [14271.278157]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.278169]         00 4f 00 c2 00 50
> [14271.278175] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.336487] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.336495] Descriptor sense data with sense descriptors (in hex):
> [14271.336498]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.336509]         00 4f 00 c2 00 50
> [14271.336515] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.428148] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.428156] Descriptor sense data with sense descriptors (in hex):
> [14271.428158]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.428170]         00 4f 00 c2 00 50
> [14271.428176] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.486485] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.486493] Descriptor sense data with sense descriptors (in hex):
> [14271.486496]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.486508]         00 4f 00 c2 00 50
> [14271.486514] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.586482] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.586489] Descriptor sense data with sense descriptors (in hex):
> [14271.586492]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.586503]         00 4f 00 c2 00 50
> [14271.586509] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.644813] sd 11:0:3:0: [sdl] Sense Key : Recovered Error
[current] [descriptor]
> [14271.644819] Descriptor sense data with sense descriptors (in hex):
> [14271.644822]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.644833]         00 4f 00 c2 00 50
> [14271.644839] sd 11:0:3:0: [sdl] Add. Sense: ATA pass through
information available
> [14271.762812] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14271.762820] Descriptor sense data with sense descriptors (in hex):
> [14271.762823]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.762834]         00 4f 00 c2 00 50
> [14271.762841] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14271.821145] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14271.821152] Descriptor sense data with sense descriptors (in hex):
> [14271.821154]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.821166]         00 4f 00 c2 00 50
> [14271.821172] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14271.912816] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14271.912824] Descriptor sense data with sense descriptors (in hex):
> [14271.912827]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.912838]         00 4f 00 c2 00 50
> [14271.912844] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14271.971152] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14271.971161] Descriptor sense data with sense descriptors (in hex):
> [14271.971163]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14271.971175]         00 4f 00 c2 00 50
> [14271.971181] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14272.071150] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14272.071157] Descriptor sense data with sense descriptors (in hex):
> [14272.071160]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14272.071172]         00 4f 00 c2 00 50
> [14272.071178] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14272.129485] sd 11:0:4:0: [sdm] Sense Key : Recovered Error
[current] [descriptor]
> [14272.129494] Descriptor sense data with sense descriptors (in hex):
> [14272.129497]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
> [14272.129508]         00 4f 00 c2 00 50
> [14272.129514] sd 11:0:4:0: [sdm] Add. Sense: ATA pass through
information available
> [14365.066847] Aborting journal on device md0:8.
> [14365.066946] __ratelimit: 246 callbacks suppressed
> [14365.066949] Buffer I/O error on device md0, logical block 854622208
> [14365.067018] lost page write due to I/O error on md0
> [14365.067023] JBD2: I/O error detected when updating journal
superblock for md0:8.
> [14382.768622] EXT4-fs error (device md0): ext4_find_entry: reading
directory #6879966 offset 0
> [14382.820264] Buffer I/O error on device md0, logical block 0
> [14382.820332] lost page write due to I/O error on md0
> [14401.997859] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891861, block=27267765
> [14401.998043] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.041639] Buffer I/O error on device md0, logical block 0
> [14402.041708] lost page write due to I/O error on md0
> [14402.042025] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892055, block=27267777
> [14402.042189] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.042337] Buffer I/O error on device md0, logical block 0
> [14402.042404] lost page write due to I/O error on md0
> [14402.042615] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891691, block=27267754
> [14402.042780] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.042927] Buffer I/O error on device md0, logical block 0
> [14402.042994] lost page write due to I/O error on md0
> [14402.043204] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891589, block=27267748
> [14402.043369] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.043514] Buffer I/O error on device md0, logical block 0
> [14402.043581] lost page write due to I/O error on md0
> [14402.045186] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892719, block=27267818
> [14402.045351] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.045500] Buffer I/O error on device md0, logical block 0
> [14402.045569] lost page write due to I/O error on md0
> [14402.061829] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891914, block=27267768
> [14402.061983] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.062117] Buffer I/O error on device md0, logical block 0
> [14402.062175] lost page write due to I/O error on md0
> [14402.062495] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891136, block=27267719
> [14402.062651] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.062793] Buffer I/O error on device md0, logical block 0
> [14402.062859] lost page write due to I/O error on md0
> [14402.063053] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6893036, block=27267838
> [14402.063217] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.063357] Buffer I/O error on device md0, logical block 0
> [14402.063423] lost page write due to I/O error on md0
> [14402.063624] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892357, block=27267796
> [14402.063793] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.063935] Buffer I/O error on device md0, logical block 0
> [14402.064001] lost page write due to I/O error on md0
> [14402.064193] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891032, block=27267713
> [14402.064355] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.064496] Buffer I/O error on device md0, logical block 0
> [14402.064561] lost page write due to I/O error on md0
> [14402.064741] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891782, block=27267760
> [14402.064906] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.065232] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892332, block=27267794
> [14402.065395] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.065714] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892755, block=27267821
> [14402.065878] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.066197] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892235, block=27267788
> [14402.066362] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.066675] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892552, block=27267808
> [14402.066840] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.067156] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892123, block=27267781
> [14402.067321] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.067635] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892256, block=27267789
> [14402.067800] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.068114] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892532, block=27267807
> [14402.068278] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.068594] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892318, block=27267793
> [14402.068758] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.069069] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892845, block=27267826
> [14402.069233] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.069543] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892980, block=27267835
> [14402.069707] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.074971] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890993, block=27267711
> [14402.075140] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.075540] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891750, block=27267758
> [14402.075686] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.076028] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891642, block=27267751
> [14402.076174] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.076543] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892605, block=27267811
> [14402.076689] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.077059] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892136, block=27267782
> [14402.077223] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.077567] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892717, block=27267818
> [14402.077732] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.078080] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890999, block=27267711
> [14402.078243] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.078593] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892362, block=27267796
> [14402.080842] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.083259] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892867, block=27267828
> [14402.083423] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.083798] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891361, block=27267734
> [14402.083963] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.084315] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892012, block=27267774
> [14402.084480] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.084852] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891626, block=27267750
> [14402.085014] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.085365] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891320, block=27267731
> [14402.085530] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.085880] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891588, block=27267748
> [14402.086044] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.086390] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892584, block=27267810
> [14402.086556] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.086901] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6892891, block=27267829
> [14402.087066] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.087416] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891118, block=27267718
> [14402.087579] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.087930] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891559, block=27267746
> [14402.088094] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.088445] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6891212, block=27267724
> [14402.088609] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.091550] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890993, block=27267711
> [14402.091718] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.106045] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890999, block=27267711
> [14402.106212] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.141662] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889579, block=27267622
> [14402.141829] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.142185] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889980, block=27267647
> [14402.142350] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.142703] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889704, block=27267630
> [14402.142868] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.143318] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889340, block=27267607
> [14402.143483] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.143826] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889563, block=27267621
> [14402.143990] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.144341] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889220, block=27267600
> [14402.144506] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.144869] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889358, block=27267608
> [14402.145034] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.145379] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889530, block=27267619
> [14402.145542] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.145890] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889721, block=27267631
> [14402.146054] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.146398] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889621, block=27267625
> [14402.146562] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.146900] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889920, block=27267643
> [14402.147047] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.147390] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889508, block=27267618
> [14402.147536] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.147869] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889673, block=27267628
> [14402.148015] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.153911] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889220, block=27267600
> [14402.154075] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.155819] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889340, block=27267607
> [14402.155987] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.261374] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876431, block=27266800
> [14402.261522] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.261981] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875468, block=27266740
> [14402.262128] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.262587] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878644, block=27266939
> [14402.262753] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.263223] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875990, block=27266773
> [14402.263388] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.263741] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875325, block=27266731
> [14402.263908] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.264259] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875779, block=27266760
> [14402.264424] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.264808] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6879881, block=27267016
> [14402.264972] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.265325] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878489, block=27266929
> [14402.265491] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.265842] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878591, block=27266935
> [14402.266005] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.266357] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876138, block=27266782
> [14402.266520] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.266876] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875310, block=27266730
> [14402.267042] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.267396] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876274, block=27266791
> [14402.267560] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.267907] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875648, block=27266751
> [14402.268071] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.268422] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876119, block=27266781
> [14402.268586] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.269056] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877808, block=27266886
> [14402.269219] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.269573] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878101, block=27266905
> [14402.269738] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.270088] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877967, block=27266896
> [14402.270264] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.270614] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877835, block=27266888
> [14402.270793] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.271146] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875370, block=27266734
> [14402.271323] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.271679] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877955, block=27266896
> [14402.271854] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.272214] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876218, block=27266787
> [14402.272391] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.272745] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875340, block=27266732
> [14402.272922] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.273281] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878031, block=27266900
> [14402.273452] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.273919] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874892, block=27266704
> [14402.274097] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.274454] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878014, block=27266899
> [14402.274628] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.274987] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6876066, block=27266778
> [14402.275146] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.275488] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6879778, block=27267010
> [14402.275646] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.275996] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6878310, block=27266918
> [14402.276151] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.276624] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877450, block=27266864
> [14402.276793] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.277148] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875397, block=27266736
> [14402.277315] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.277778] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6877004, block=27266836
> [14402.277943] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.278295] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875606, block=27266749
> [14402.280543] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.283306] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874892, block=27266704
> [14402.283472] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.285354] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875310, block=27266730
> [14402.285519] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.302533] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874640, block=27266688
> [14402.302698] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.304480] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874640, block=27266688
> [14402.304629] EXT4-fs (md0): previous I/O error to superblock detected
> [14402.752437] EXT4-fs error (device md0): ext4_journal_start_sb:
Detected aborted journal
> [14402.752606] EXT4-fs (md0): Remounting filesystem read-only
> [14419.267133] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874640, block=27266688
> [14419.297937] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6874892, block=27266704
> [14419.301517] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6875310, block=27266730
> [14419.332861] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889220, block=27267600
> [14419.335590] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6889340, block=27267607
> [14419.341744] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890993, block=27267711
> [14419.343458] EXT4-fs error (device md0): __ext4_get_inode_loc:
unable to read inode block - inode=6890999, block=27267711


Richard Scobie wrote:
> Andrew Dunn wrote:
>
>> Do you think that the controller is dropping out? I know that I have 4
>> drives on one controller (AOC-USAS-L8i) and 5 drives on the other
>> controller (SAME make/model). but I think they are sequentially
>> connected... as in sd[efghi] should be on one device and sd[jklm] should
>> be on the other... any easy way to verify?
>
> If you are running smartd, cease doing so and do not use the smartctl
> command on drives attached to these controllers - use causes drives to
> be offlined.
>
> It appears the smartctl is broken with LSISAS 1068E based controllers.
>
> See:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=452389
>
> and
>
> http://marc.info/?l=linux-scsi&m=125673590221135&w=2
>
> Regards,
>
> Richard
>

-- 
Andrew Dunn
http://agdunn.net