WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Vladislav Bolkhovitin]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

Hi,

During recent debugging session of my SCSI target SCST 
(http://scst.sf.net) I noticed many

WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

messages in kernel log on the initiator. I attached the full log of 
several of them.

My target was buggy and I was working on fixing it, but I suppose Linux 
should handle such failures more gracefully. In all the cases the target 
had one type of failure: it "ate" a SCSI command and never returned 
result of it.

Low level drivers were open-iscsi and qla2xxx.

Vlad

[-- Attachment #2: k.tar.bz2 --]
[-- Type: application/x-bzip, Size: 42073 bytes --]

[-- Attachment #3: kernel --]
[-- Type: text/plain, Size: 6140 bytes --]

Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 816
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_get_inode_loc: unable to read inode block - inode=2, block=316
Oct 16 21:58:54 inif kernel: ------------[ cut here ]------------
Oct 16 21:58:54 inif kernel: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
Oct 16 21:58:54 inif kernel: Modules linked in: crc32c crypto_hash libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi firmware_class scsi_transport_fc tcp_highspeed w83627hf hwmon_vid eeprom adm1021 hwmon binfmt_misc dm_mirror dm_log dm_mod pci_slot battery ac ehci_hcd uhci_hcd sg e1000 pcspkr i2c_i801 i2c_core ide_cd_mod cdrom e7xxx_edac button usbcore [last unloaded: qla2xxx]
Oct 16 21:58:54 inif kernel: Pid: 19064, comm: pam_console_app Not tainted 2.6.27-ini #3
Oct 16 21:58:54 inif kernel:  [<b0116065>] warn_on_slowpath+0x4d/0x66
Oct 16 21:58:54 inif kernel:  [<b01270bd>] enqueue_hrtimer+0x72/0xea
Oct 16 21:58:54 inif kernel:  [<b0112adb>] hrtick_start_fair+0x8e/0xd9
Oct 16 21:58:54 inif kernel:  [<b0112dc3>] pick_next_task_fair+0x8c/0xa2
Oct 16 21:58:54 inif kernel:  [<b03098fa>] schedule+0x173/0x3e9
Oct 16 21:58:54 inif kernel:  [<b017a205>] mark_buffer_dirty+0x51/0x66
Oct 16 21:58:54 inif kernel:  [<b01a5abc>] ext3_commit_super+0x3a/0x4e
Oct 16 21:58:54 inif kernel:  [<b01a6603>] ext3_handle_error+0x62/0xaa
Oct 16 21:58:54 inif kernel:  [<b0116d25>] printk+0x1b/0x1f
Oct 16 21:58:54 inif kernel:  [<b01a66fd>] ext3_error+0x55/0x5a
Oct 16 21:58:54 inif kernel:  [<b019e90f>] __ext3_get_inode_loc+0x253/0x2b9
Oct 16 21:58:54 inif kernel:  [<b019e99f>] ext3_reserve_inode_write+0x19/0x5f
Oct 16 21:58:54 inif kernel:  [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel:  [<b019e9ff>] ext3_mark_inode_dirty+0x1a/0x30
Oct 16 21:58:54 inif kernel:  [<b01ae8ec>] journal_start+0x80/0xa9
Oct 16 21:58:54 inif kernel:  [<b01a16d6>] ext3_dirty_inode+0x65/0x6e
Oct 16 21:58:54 inif kernel:  [<b0176b31>] __mark_inode_dirty+0x24/0x133
Oct 16 21:58:54 inif kernel:  [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel:  [<b016eb53>] touch_atime+0xa1/0xee
Oct 16 21:58:54 inif kernel:  [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel:  [<b0168077>] vfs_readdir+0x7c/0x7e
Oct 16 21:58:54 inif kernel:  [<b01680dc>] sys_getdents64+0x63/0xa5
Oct 16 21:58:54 inif kernel:  [<b0102cb9>] sysenter_do_call+0x12/0x21
Oct 16 21:58:54 inif kernel:  =======================
Oct 16 21:58:54 inif kernel: ---[ end trace ddb1531dc775930c ]---
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf) in ext3_reserve_inode_write: IO failure
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif last message repeated 157 times
Oct 16 21:59:05 inif kernel: journal_bmap: journal block not found at offset 859 on sdf
Oct 16 21:59:05 inif kernel: Aborting journal on device sdf.
Oct 16 21:59:05 inif kernel: __ratelimit: 158 callbacks suppressed
Oct 16 21:59:05 inif kernel: Buffer I/O error on device sdf, logical block 822
Oct 16 21:59:05 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:59:05 inif kernel: journal commit I/O error
Oct 16 21:59:05 inif kernel: sd 12:0:0:0: [sdb] Synchronizing SCSI cache
Oct 16 21:59:05 inif kernel: sd 12:0:0:2: [sdd] Synchronizing SCSI cache
Oct 16 21:59:05 inif kernel: sd 12:0:0:3: [sde] Synchronizing SCSI cache
Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:59:12 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:59:12 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:59:12 inif kernel: ext3_abort called.
Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_journal_start_sb: Detected aborted journal
Oct 16 21:59:12 inif kernel: Remounting filesystem read-only
Oct 16 21:59:44 inif kernel: Buffer I/O error on device sdf, logical block 822
Oct 16 21:59:44 inif kernel: lost page write due to I/O error on sdf

Re: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Nick Piggin]

On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> Hi,
>
> During recent debugging session of my SCSI target SCST
> (http://scst.sf.net) I noticed many
>
> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>
> messages in kernel log on the initiator. I attached the full log of
> several of them.
>
> My target was buggy and I was working on fixing it, but I suppose Linux
> should handle such failures more gracefully. In all the cases the target
> had one type of failure: it "ate" a SCSI command and never returned
> result of it.

Right. This is one of the warnings I see in my fault-injection testing.
It is fixed by my patch to clean up and improve the page and buffer
error handling in the vm/fs.

Re: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Vladislav Bolkhovitin]

Nick Piggin wrote:
> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>> Hi,
>>
>> During recent debugging session of my SCSI target SCST
>> (http://scst.sf.net) I noticed many
>>
>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>
>> messages in kernel log on the initiator. I attached the full log of
>> several of them.
>>
>> My target was buggy and I was working on fixing it, but I suppose Linux
>> should handle such failures more gracefully. In all the cases the target
>> had one type of failure: it "ate" a SCSI command and never returned
>> result of it.
> 
> Right. This is one of the warnings I see in my fault-injection testing.
> It is fixed by my patch to clean up and improve the page and buffer
> error handling in the vm/fs.

Can you specify which patch you referring? Is it in 2.6.27?

Re: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Nick Piggin]

On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
> > On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> >> Hi,
> >>
> >> During recent debugging session of my SCSI target SCST
> >> (http://scst.sf.net) I noticed many
> >>
> >> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
> >>
> >> messages in kernel log on the initiator. I attached the full log of
> >> several of them.
> >>
> >> My target was buggy and I was working on fixing it, but I suppose Linux
> >> should handle such failures more gracefully. In all the cases the target
> >> had one type of failure: it "ate" a SCSI command and never returned
> >> result of it.
> >
> > Right. This is one of the warnings I see in my fault-injection testing.
> > It is fixed by my patch to clean up and improve the page and buffer
> > error handling in the vm/fs.
>
> Can you specify which patch you referring? Is it in 2.6.27?

It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.

Re: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Vladislav Bolkhovitin]

Nick Piggin wrote:
> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
>> Nick Piggin wrote:
>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>>>> Hi,
>>>>
>>>> During recent debugging session of my SCSI target SCST
>>>> (http://scst.sf.net) I noticed many
>>>>
>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>>>
>>>> messages in kernel log on the initiator. I attached the full log of
>>>> several of them.
>>>>
>>>> My target was buggy and I was working on fixing it, but I suppose Linux
>>>> should handle such failures more gracefully. In all the cases the target
>>>> had one type of failure: it "ate" a SCSI command and never returned
>>>> result of it.
>>> Right. This is one of the warnings I see in my fault-injection testing.
>>> It is fixed by my patch to clean up and improve the page and buffer
>>> error handling in the vm/fs.
>> Can you specify which patch you referring? Is it in 2.6.27?
> 
> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.

I see. I'm looking forward to see it in 2.6.28 or .29. This is really a 
needed work.

BTW, have you even seen in your fault-injection testing that after 
receiving a failure from a SCSI device during heavy load ext3 file 
system mounted on it gets corrupted and journal replay on remount 
doesn't repair it, only manual e2fsck helps? I've many times seen that, 
including cases when the target was remaining up and fully functional. 
See, e.g., "MOANING MODE ON" part in 
http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked 
that case since then, although I see such corruptions quite often. But 
in all them I can't so clearly say that it isn't a target's failure.

Vlad

Re: FS corruption after I/O errors

[Vladislav Bolkhovitin]

[-- Attachment #1: Type: text/plain, Size: 2314 bytes --]

Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
>> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
>>> Nick Piggin wrote:
>>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>>>>> Hi,
>>>>>
>>>>> During recent debugging session of my SCSI target SCST
>>>>> (http://scst.sf.net) I noticed many
>>>>>
>>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>>>>
>>>>> messages in kernel log on the initiator. I attached the full log of
>>>>> several of them.
>>>>>
>>>>> My target was buggy and I was working on fixing it, but I suppose Linux
>>>>> should handle such failures more gracefully. In all the cases the target
>>>>> had one type of failure: it "ate" a SCSI command and never returned
>>>>> result of it.
>>>> Right. This is one of the warnings I see in my fault-injection testing.
>>>> It is fixed by my patch to clean up and improve the page and buffer
>>>> error handling in the vm/fs.
>>> Can you specify which patch you referring? Is it in 2.6.27?
>> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
> 
> I see. I'm looking forward to see it in 2.6.28 or .29. This is really a 
> needed work.
> 
> BTW, have you even seen in your fault-injection testing that after 
> receiving a failure from a SCSI device during heavy load ext3 file 
> system mounted on it gets corrupted and journal replay on remount 
> doesn't repair it, only manual e2fsck helps? I've many times seen that, 
> including cases when the target was remaining up and fully functional. 
> See, e.g., "MOANING MODE ON" part in 
> http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked 
> that case since then, although I see such corruptions quite often. But 
> in all them I can't so clearly say that it isn't a target's failure.

I've just checked it with 2.6.27. The situation greatly improved and 
dbench was able to complete several runs under constant TASK_ABORTED 
"bombarding" (TASK RESET task management commands using "sg_reset -b" 
each 31 seconds from another "connection" to that device via qla2xxx 
initiator driver. You can see those resets in the attached log). But 
when then I unmounted the affected partition, e2fsck found errors on it. 
See attachments for details. The target all the times was fine and 
completely healthy.



[-- Attachment #2: dbench --]
[-- Type: text/plain, Size: 17885 bytes --]

[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks
[root@inif w]# mount -t ext3 /dev/sdc /mnt
[root@inif dbench-mod]# while true; do ./dbench 50; done
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++**************************************************
Throughput 27.5109 MB/sec (NB=34.3887 MB/sec  275.109 MBit/sec)
Average latency 7.166755 ms
Maximum latency 22972.843000 ms
Latency > 2 sec. 0.091621 %
Latency > 10 sec. 0.016009 %
50 clients started
............................................................................................................write failed on handle 4290
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279
........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++**************************************************
Throughput 27.5921 MB/sec (NB=34.4902 MB/sec  275.921 MBit/sec)
Average latency 7.097532 ms
Maximum latency 20892.153000 ms
Latency > 2 sec. 0.087833 %
Latency > 10 sec. 0.016437 %
50 clients started
....................................................................................write failed on handle 4290
..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists)
(4730) nb_close: handle 4328 was not open
(4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists)
(4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists)
(4752) nb_read: handle 4330 was not open size=24 ofs=2156
(4754) nb_read: handle 4330 was not open size=276 ofs=2139
(4755) nb_read: handle 4330 was not open size=14 ofs=2136
(4757) nb_read: handle 4330 was not open size=7 ofs=2136
(4759) nb_read: handle 4330 was not open size=12 ofs=2138
(4761) nb_read: handle 4330 was not open size=8 ofs=2137
(4763) nb_read: handle 4330 was not open size=15 ofs=2135
(4765) nb_read: handle 4330 was not open size=16 ofs=0
(4767) nb_read: handle 4330 was not open size=2048 ofs=16
(4768) nb_read: handle 4330 was not open size=2048 ofs=2064
(4769) nb_read: handle 4330 was not open size=2048 ofs=257
(4770) nb_read: handle 4330 was not open size=2048 ofs=385
(4771) nb_read: handle 4330 was not open size=2048 ofs=513
(4772) nb_read: handle 4330 was not open size=2048 ofs=641
(4773) nb_read: handle 4330 was not open size=2048 ofs=769
(4774) nb_read: handle 4330 was not open size=2048 ofs=897
(4775) nb_read: handle 4330 was not open size=2048 ofs=1025
(4776) nb_read: handle 4330 was not open size=2048 ofs=1153
(4777) nb_read: handle 4330 was not open size=2048 ofs=1281
(4778) nb_read: handle 4330 was not open size=2048 ofs=1409
(4779) nb_read: handle 4330 was not open size=408 ofs=1537
..(4899) nb_close: handle 4330 was not open
(4964) nb_read: handle 4329 was not open size=1386 ofs=2157
(4971) nb_read: handle 4329 was not open size=4096 ofs=2157
(4972) nb_read: handle 4329 was not open size=4096 ofs=6253
(4973) nb_read: handle 4329 was not open size=4096 ofs=10349
(4974) nb_read: handle 4329 was not open size=4096 ofs=14445
(4975) nb_read: handle 4329 was not open size=4096 ofs=18541
(4976) nb_read: handle 4329 was not open size=256 ofs=22637
(4977) nb_read: handle 4329 was not open size=4096 ofs=22893
(4978) nb_read: handle 4329 was not open size=4096 ofs=26989
(4979) nb_read: handle 4329 was not open size=4096 ofs=31085
(4980) nb_read: handle 4329 was not open size=4096 ofs=35181
(4981) nb_read: handle 4329 was not open size=4096 ofs=39277
(4982) nb_read: handle 4329 was not open size=4096 ofs=43373
(4983) nb_read: handle 4329 was not open size=4096 ofs=47469
(4986) nb_read: handle 4329 was not open size=1024 ofs=2157
(4987) nb_read: handle 4329 was not open size=4688 ofs=1667
(4988) nb_read: handle 4329 was not open size=3516 ofs=6355
(4989) nb_read: handle 4329 was not open size=16384 ofs=9871
(4990) nb_read: handle 4329 was not open size=3771 ofs=26255
(4991) nb_read: handle 4329 was not open size=2048 ofs=30026
(4992) nb_read: handle 4329 was not open size=1024 ofs=32074
(4993) nb_read: handle 4329 was not open size=6652 ofs=33098
(4994) nb_read: handle 4329 was not open size=16384 ofs=39750
(4995) nb_read: handle 4329 was not open size=10496 ofs=56134
(4996) nb_read: handle 4329 was not open size=1784 ofs=66630
(4997) nb_read: handle 4329 was not open size=90 ofs=68414
(4998) nb_read: handle 4329 was not open size=273 ofs=68504
(4999) nb_read: handle 4329 was not open size=1600 ofs=68777
(5000) nb_read: handle 4329 was not open size=640 ofs=70377
(5001) nb_read: handle 4329 was not open size=1280 ofs=71017
(5002) nb_read: handle 4329 was not open size=8464 ofs=72297
(5003) nb_read: handle 4329 was not open size=158 ofs=80761
(5004) nb_read: handle 4329 was not open size=912 ofs=80919
(5005) nb_read: handle 4329 was not open size=1980 ofs=81831
(5006) nb_read: handle 4329 was not open size=2256 ofs=83811
(5007) nb_read: handle 4329 was not open size=400 ofs=86067
(5008) nb_read: handle 4329 was not open size=252 ofs=86467
(5009) nb_read: handle 4329 was not open size=42 ofs=86719
(5011) nb_read: handle 4329 was not open size=725 ofs=86761
(5012) nb_read: handle 4329 was not open size=512 ofs=2009
(5013) nb_read: handle 4329 was not open size=9 ofs=3895
(5015) nb_read: handle 4329 was not open size=46 ofs=3781
(5017) nb_read: handle 4329 was not open size=92 ofs=4059
(5018) nb_read: handle 4329 was not open size=92 ofs=3781
(5019) nb_read: handle 4329 was not open size=184 ofs=3986
(5020) nb_read: handle 4329 was not open size=16384 ofs=3987
(5021) nb_read: handle 4329 was not open size=16384 ofs=3902
(5022) nb_read: handle 4329 was not open size=10240 ofs=3781
(5023) nb_read: handle 4329 was not open size=16384 ofs=3821
(5024) nb_read: handle 4329 was not open size=935 ofs=3891
(5025) nb_read: handle 4329 was not open size=10240 ofs=4060
(5026) nb_read: handle 4329 was not open size=184 ofs=4059
(5027) nb_read: handle 4329 was not open size=184 ofs=256
(5028) nb_read: handle 4329 was not open size=1026 ofs=256
(5029) nb_read: handle 4329 was not open size=326 ofs=3901
(5030) nb_read: handle 4329 was not open size=652 ofs=3899
(5031) nb_read: handle 4329 was not open size=2 ofs=256
(5033) nb_read: handle 4329 was not open size=4948 ofs=258
(5034) nb_close: handle 4329 was not open
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++**************************************************
Throughput 27.8199 MB/sec (NB=34.7749 MB/sec  278.199 MBit/sec)
Average latency 6.940010 ms
Maximum latency 18993.291000 ms
Latency > 2 sec. 0.081048 %
Latency > 10 sec. 0.015581 %
50 clients started
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++**************************************************
Throughput 27.0223 MB/sec (NB=33.7779 MB/sec  270.223 MBit/sec)
Average latency 6.739319 ms
Maximum latency 18699.530000 ms
Latency > 2 sec. 0.090834 %
Latency > 10 sec. 0.013650 %
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++**************************************************
Throughput 27.2861 MB/sec (NB=34.1077 MB/sec  272.861 MBit/sec)
Average latency 7.051013 ms
Maximum latency 23801.490000 ms
Latency > 2 sec. 0.091263 %
Latency > 10 sec. 0.014722 %
50 clients started
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
[root@inif w]# umount /mnt
[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(1250434--1250435) -(1273360--1273370)
Fix? yes

Free blocks count wrong for group #38 (32253, counted=32266).
Fix? yes

Free blocks count wrong (1003250, counted=1003263).
Fix? yes


/dev/sdc: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocks

[-- Attachment #3: kernel --]
[-- Type: text/plain, Size: 38110 bytes --]

Nov  1 16:27:35 inif kernel: kjournald starting.  Commit interval 5 seconds
Nov  1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal
Nov  1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov  1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352
Nov  1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808
Nov  1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0
Nov  1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432
Nov  1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938
Nov  1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088
Nov  1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456
Nov  1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792
Nov  1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240
Nov  1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720
Nov  1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752
Nov  1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288
Nov  1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795
Nov  1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512
Nov  1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536
Nov  1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784
Nov  1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352
Nov  1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776
Nov  1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0
Nov  1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544
Nov  1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952
Nov  1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480
Nov  1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819
Nov  1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840
Nov  1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239
Nov  1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472
Nov  1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880
Nov  1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768
Nov  1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392
Nov  1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0
Nov  1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128
Nov  1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270
Nov  1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064
Nov  1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504
Nov  1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322
Nov  1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400
Nov  1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376
Nov  1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181
Nov  1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov  1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192
Nov  1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872
Nov  1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov  1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov  1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov  1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov  1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760
Nov  1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729
Nov  1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov  1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov  1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864
Nov  1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded

Re: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

[Nick Piggin]

On Saturday 01 November 2008 05:11, Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
> > On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
> >> Nick Piggin wrote:
> >>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> >>>> Hi,
> >>>>
> >>>> During recent debugging session of my SCSI target SCST
> >>>> (http://scst.sf.net) I noticed many
> >>>>
> >>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
> >>>>
> >>>> messages in kernel log on the initiator. I attached the full log of
> >>>> several of them.
> >>>>
> >>>> My target was buggy and I was working on fixing it, but I suppose
> >>>> Linux should handle such failures more gracefully. In all the cases
> >>>> the target had one type of failure: it "ate" a SCSI command and never
> >>>> returned result of it.
> >>>
> >>> Right. This is one of the warnings I see in my fault-injection testing.
> >>> It is fixed by my patch to clean up and improve the page and buffer
> >>> error handling in the vm/fs.
> >>
> >> Can you specify which patch you referring? Is it in 2.6.27?
> >
> > It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
>
> I see. I'm looking forward to see it in 2.6.28 or .29. This is really a
> needed work.

Hopefully. Unfortunately it doesn't exactly make the filesystems
themselves more robust against failure. That needs to be done on
a case by case basis.


> BTW, have you even seen in your fault-injection testing that after
> receiving a failure from a SCSI device during heavy load ext3 file
> system mounted on it gets corrupted and journal replay on remount
> doesn't repair it, only manual e2fsck helps? I've many times seen that,
> including cases when the target was remaining up and fully functional.
> See, e.g., "MOANING MODE ON" part in
> http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked
> that case since then, although I see such corruptions quite often. But
> in all them I can't so clearly say that it isn't a target's failure.

I haven't seen that, but I'm not exactly testing for filesystem
robustness to errors, but more of core vm/fs layer robustness, so
I'm mainly trying to inject errors into data portion of inodes.

I think the ext3 development list should be interested in your
report.

Thanks,
Nick