All of lore.kernel.org
 help / color / mirror / Atom feed
* Input/Output error
@ 2011-02-22 15:44 Srinivasan T
  2011-02-22 20:21 ` Eric Sandeen
  2011-02-23  0:47 ` Dave Chinner
  0 siblings, 2 replies; 18+ messages in thread
From: Srinivasan T @ 2011-02-22 15:44 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 3134 bytes --]

Hi,

We are running an C++ application in AWS EC2 instance (CentOS 5.4) 
mounted with an EBS Volume (say /mymountpoint). We do more simultaneous 
writes to the EBS Volume from our application. But at some point we get 
'ERROR: Input/output error'. After this, 'ls -l /mymountpoint' command 
itself fails with the i/o error. The filesystem which we use for the EBS 
Volume is xfs.

I unmounted the drive and done xfs_check and again mounted the drive. 
Now, everything seems to be working fine. But the issue still persists 
everytime when we do simultaneous writes.

I believe the following details will be useful,

[root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
CentOS release 5.4 (Final)

[root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1     ext3   1310720  107566 1203154    9% /
/dev/sdb      ext3   19546112      11 19546101    1% /mnt
none         tmpfs    186059       1  186058    1% /dev/shm
/dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint

[root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 
12:39:36 EST 2008 i686 i686 i386 GNU/Linux

Output of dmesg :

SGI XFS with ACLs, security attributes, large block numbers, no debug 
enabled
SGI XFS Quota Management subsystem
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Ending clean XFS mount for filesystem: sdh
Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of 
file fs/xfs/xfs_trans.c.  Caller 0xee201944
  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
  [<c1078d88>] vfs_rename+0x2c3/0x441
  [<c107a77f>] sys_renameat+0x15a/0x1b4
  [<c1074b7f>] sys_stat64+0xf/0x23
  [<c1072d3b>] __fput+0x140/0x16a
  [<c10841ee>] mntput_no_expire+0x11/0x6a
  [<c107a800>] sys_rename+0x27/0x2b
  [<c1005688>] syscall_call+0x7/0xb
  =======================
xfs_force_shutdown(sdh,0x8) called from line 1139 of file 
fs/xfs/xfs_trans.c.  Return address = 0xee217778
Filesystem "sdh": Corruption of in-memory data detected.  Shutting down 
filesystem: sdh
Please umount the filesystem, and rectify the problem(s)
I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       
("xfs_trans_read_buf") error 5 buf count 512
I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       
("xfs_trans_read_buf") error 5 buf count 512
xfs_force_shutdown(sdh,0x1) called from line 423 of file 
fs/xfs/xfs_rw.c.  Return address = 0xee217778
xfs_force_shutdown(sdh,0x1) called from line 423 of file 
fs/xfs/xfs_rw.c.  Return address = 0xee217778
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Starting XFS recovery on filesystem: sdh (logdev: internal)
Ending XFS recovery on filesystem: sdh (logdev: internal)

The XFS utilities are in v2.9.4

Any help would be appreciated.

Regards,
Srinivasan

[-- Attachment #1.2: Type: text/html, Size: 4866 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/Output error
  2011-02-22 15:44 Input/Output error Srinivasan T
@ 2011-02-22 20:21 ` Eric Sandeen
  2011-02-23  6:33   ` Srinivasan T
  2011-02-23  0:47 ` Dave Chinner
  1 sibling, 1 reply; 18+ messages in thread
From: Eric Sandeen @ 2011-02-22 20:21 UTC (permalink / raw)
  To: Srinivasan T; +Cc: xfs

On 2/22/11 9:44 AM, Srinivasan T wrote:
> Hi,
> 
> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
> mounted with an EBS Volume (say /mymountpoint). We do more
> simultaneous writes to the EBS Volume from our application. But at
> some point we get 'ERROR: Input/output error'. After this, 'ls -l
> /mymountpoint' command itself fails with the i/o error. The
> filesystem which we use for the EBS Volume is xfs.
> 
> I unmounted the drive and done xfs_check and again mounted the drive.

xfs_check is read-only, FWIW, a task best handled these days
by xfs_repair -n.

> Now, everything seems to be working fine. But the issue still
> persists everytime when we do simultaneous writes.> 
> I believe the following details will be useful,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
> CentOS release 5.4 (Final)

is quota in use?  Might you be running out of space on the fs?

Sadly I'm not even sure what xfs you might be running; there were centos kmod
rpms of older xfs for a while, and then more recent kernels have xfs.ko
built in, because RHEL5 started with official support for XFS later on.

line 1138 of file fs/xfs/xfs_trans.c does not line up with current RHEL5.

"modinfo xfs" might tell us at least where the xfs module was found,
and from there know where it came from.

But, xfs support on RHEL is best handled by Red Hat, I'm afraid.  And if this
is an old xfs kmod (which *cough* I did for centos years ago) it's not going
to be well supported by anyone.

-if- you have xfs-kmod installed, and a kernel rpm which contains xfs.ko
itself, I'd suggest trying again with the latter.

Barring all that, perhaps this is a known problem more obvious to another
person on this list ...

-Eric

> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda1     ext3   1310720  107566 1203154    9% /
> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
> none         tmpfs    186059       1  186058    1% /dev/shm
> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
> 
> Output of dmesg :
> 
> SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
> SGI XFS Quota Management subsystem
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Ending clean XFS mount for filesystem: sdh
> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>  [<c1078d88>] vfs_rename+0x2c3/0x441
>  [<c107a77f>] sys_renameat+0x15a/0x1b4
>  [<c1074b7f>] sys_stat64+0xf/0x23
>  [<c1072d3b>] __fput+0x140/0x16a
>  [<c10841ee>] mntput_no_expire+0x11/0x6a
>  [<c107a800>] sys_rename+0x27/0x2b
>  [<c1005688>] syscall_call+0x7/0xb
>  =======================
> xfs_force_shutdown(sdh,0x8) called from line 1139 of file fs/xfs/xfs_trans.c.  Return address = 0xee217778
> Filesystem "sdh": Corruption of in-memory data detected.  Shutting down filesystem: sdh
> Please umount the filesystem, and rectify the problem(s)
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       ("xfs_trans_read_buf") error 5 buf count 512
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       ("xfs_trans_read_buf") error 5 buf count 512
> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Starting XFS recovery on filesystem: sdh (logdev: internal)
> Ending XFS recovery on filesystem: sdh (logdev: internal)
> 
> The XFS utilities are in v2.9.4
> 
> Any help would be appreciated.
> 
> Regards,
> Srinivasan
> 
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/Output error
  2011-02-22 15:44 Input/Output error Srinivasan T
  2011-02-22 20:21 ` Eric Sandeen
@ 2011-02-23  0:47 ` Dave Chinner
  1 sibling, 0 replies; 18+ messages in thread
From: Dave Chinner @ 2011-02-23  0:47 UTC (permalink / raw)
  To: Srinivasan T; +Cc: xfs

On Tue, Feb 22, 2011 at 09:14:24PM +0530, Srinivasan T wrote:
> Hi,
> 
> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
> mounted with an EBS Volume (say /mymountpoint). We do more
> simultaneous writes to the EBS Volume from our application. But at
> some point we get 'ERROR: Input/output error'. After this, 'ls -l
> /mymountpoint' command itself fails with the i/o error. The
> filesystem which we use for the EBS Volume is xfs.
> 
> I unmounted the drive and done xfs_check and again mounted the
> drive. Now, everything seems to be working fine. But the issue still
> persists everytime when we do simultaneous writes.
> 
> I believe the following details will be useful,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
> CentOS release 5.4 (Final)
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda1     ext3   1310720  107566 1203154    9% /
> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
> none         tmpfs    186059       1  186058    1% /dev/shm
> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15
> 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
> 
> Output of dmesg :
> 
> SGI XFS with ACLs, security attributes, large block numbers, no
> debug enabled
> SGI XFS Quota Management subsystem
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Ending clean XFS mount for filesystem: sdh
> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138
> of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>  [<c1078d88>] vfs_rename+0x2c3/0x441
>  [<c107a77f>] sys_renameat+0x15a/0x1b4
>  [<c1074b7f>] sys_stat64+0xf/0x23
>  [<c1072d3b>] __fput+0x140/0x16a
>  [<c10841ee>] mntput_no_expire+0x11/0x6a
>  [<c107a800>] sys_rename+0x27/0x2b
>  [<c1005688>] syscall_call+0x7/0xb
>  =======================
> xfs_force_shutdown(sdh,0x8) called from line 1139 of file
> fs/xfs/xfs_trans.c.  Return address = 0xee217778
> Filesystem "sdh": Corruption of in-memory data detected.  Shutting
> down filesystem: sdh
> Please umount the filesystem, and rectify the problem(s)
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001
> ("xfs_trans_read_buf") error 5 buf count 512
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001
> ("xfs_trans_read_buf") error 5 buf count 512

Looks like an IO error reading the AGF during a transaction. That
would indicate a problem with your storage, not XFS.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/Output error
  2011-02-22 20:21 ` Eric Sandeen
@ 2011-02-23  6:33   ` Srinivasan T
  2011-02-23 16:32     ` Eric Sandeen
  2011-02-23 21:55     ` Dave Chinner
  0 siblings, 2 replies; 18+ messages in thread
From: Srinivasan T @ 2011-02-23  6:33 UTC (permalink / raw)
  To: Eric Sandeen; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 5529 bytes --]

Hi Eric,

There is enough disk space in the fs.

[root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  2.9G  6.6G  31% /
/dev/sdb              147G  188M  140G   1% /mnt
none                  854M     0  854M   0% /dev/shm
/dev/sdh              5.0G   25M  5.0G   1% /mymountpoint

Also, the output of modinfo is as follows,

[root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko
license:        GPL
description:    SGI XFS with ACLs, security attributes, large block 
numbers, no debug enabled
author:         Silicon Graphics, Inc.
srcversion:     C7114C18263E3067C64F2BC
depends:
vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS

I have installed xfs with yum by using the following command,

yum install xfsprogs

I think I have installed the kernel rpm xfs ko. Can you confirm if this 
is correct.

Regards,
Srinivasan


On 02/23/2011 01:51 AM, Eric Sandeen wrote:
> On 2/22/11 9:44 AM, Srinivasan T wrote:
>> Hi,
>>
>> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
>> mounted with an EBS Volume (say /mymountpoint). We do more
>> simultaneous writes to the EBS Volume from our application. But at
>> some point we get 'ERROR: Input/output error'. After this, 'ls -l
>> /mymountpoint' command itself fails with the i/o error. The
>> filesystem which we use for the EBS Volume is xfs.
>>
>> I unmounted the drive and done xfs_check and again mounted the drive.
> xfs_check is read-only, FWIW, a task best handled these days
> by xfs_repair -n.
>
>> Now, everything seems to be working fine. But the issue still
>> persists everytime when we do simultaneous writes.>
>> I believe the following details will be useful,
>>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
>> CentOS release 5.4 (Final)
> is quota in use?  Might you be running out of space on the fs?
>
> Sadly I'm not even sure what xfs you might be running; there were centos kmod
> rpms of older xfs for a while, and then more recent kernels have xfs.ko
> built in, because RHEL5 started with official support for XFS later on.
>
> line 1138 of file fs/xfs/xfs_trans.c does not line up with current RHEL5.
>
> "modinfo xfs" might tell us at least where the xfs module was found,
> and from there know where it came from.
>
> But, xfs support on RHEL is best handled by Red Hat, I'm afraid.  And if this
> is an old xfs kmod (which *cough* I did for centos years ago) it's not going
> to be well supported by anyone.
>
> -if- you have xfs-kmod installed, and a kernel rpm which contains xfs.ko
> itself, I'd suggest trying again with the latter.
>
> Barring all that, perhaps this is a known problem more obvious to another
> person on this list ...
>
> -Eric
>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
>> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
>> /dev/sda1     ext3   1310720  107566 1203154    9% /
>> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
>> none         tmpfs    186059       1  186058    1% /dev/shm
>> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
>>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
>> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
>>
>> Output of dmesg :
>>
>> SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
>> SGI XFS Quota Management subsystem
>> Filesystem "sdh": Disabling barriers, not supported by the underlying device
>> XFS mounting filesystem sdh
>> Ending clean XFS mount for filesystem: sdh
>> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>>   [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>>   [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>>   [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>>   [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>>   [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>>   [<c1078d88>] vfs_rename+0x2c3/0x441
>>   [<c107a77f>] sys_renameat+0x15a/0x1b4
>>   [<c1074b7f>] sys_stat64+0xf/0x23
>>   [<c1072d3b>] __fput+0x140/0x16a
>>   [<c10841ee>] mntput_no_expire+0x11/0x6a
>>   [<c107a800>] sys_rename+0x27/0x2b
>>   [<c1005688>] syscall_call+0x7/0xb
>>   =======================
>> xfs_force_shutdown(sdh,0x8) called from line 1139 of file fs/xfs/xfs_trans.c.  Return address = 0xee217778
>> Filesystem "sdh": Corruption of in-memory data detected.  Shutting down filesystem: sdh
>> Please umount the filesystem, and rectify the problem(s)
>> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       ("xfs_trans_read_buf") error 5 buf count 512
>> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       ("xfs_trans_read_buf") error 5 buf count 512
>> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
>> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
>> Filesystem "sdh": Disabling barriers, not supported by the underlying device
>> XFS mounting filesystem sdh
>> Starting XFS recovery on filesystem: sdh (logdev: internal)
>> Ending XFS recovery on filesystem: sdh (logdev: internal)
>>
>> The XFS utilities are in v2.9.4
>>
>> Any help would be appreciated.
>>
>> Regards,
>> Srinivasan
>>
>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs


[-- Attachment #1.2: Type: text/html, Size: 7035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/Output error
  2011-02-23  6:33   ` Srinivasan T
@ 2011-02-23 16:32     ` Eric Sandeen
  2011-02-23 21:55     ` Dave Chinner
  1 sibling, 0 replies; 18+ messages in thread
From: Eric Sandeen @ 2011-02-23 16:32 UTC (permalink / raw)
  To: Srinivasan T; +Cc: xfs

On 2/23/11 12:33 AM, Srinivasan T wrote:
> Hi Eric,
> 
> There is enough disk space in the fs.
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.9G  6.6G  31% /
> /dev/sdb              147G  188M  140G   1% /mnt
> none                  854M     0  854M   0% /dev/shm
> /dev/sdh              5.0G   25M  5.0G   1% /mymountpoint
> 
> Also, the output of modinfo is as follows,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
> filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko

so a fedora 8 xen kernel on a centos 5.4 machine... you are pretty far
from the supported realm here, I'm afraid.

Can you check your logs for IO errors as dave suggested; otherwise, try
upstream or at -least- an official RHEL5 install.

-Eric

> license:        GPL
> description:    SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
> author:         Silicon Graphics, Inc.
> srcversion:     C7114C18263E3067C64F2BC
> depends:
> vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS
> 
> I have installed xfs with yum by using the following command,
> 
> yum install xfsprogs
> 
> I think I have installed the kernel rpm xfs ko. Can you confirm if this is correct.
> 
> Regards,
> Srinivasan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/Output error
  2011-02-23  6:33   ` Srinivasan T
  2011-02-23 16:32     ` Eric Sandeen
@ 2011-02-23 21:55     ` Dave Chinner
  1 sibling, 0 replies; 18+ messages in thread
From: Dave Chinner @ 2011-02-23 21:55 UTC (permalink / raw)
  To: Srinivasan T; +Cc: Eric Sandeen, xfs

On Wed, Feb 23, 2011 at 12:03:12PM +0530, Srinivasan T wrote:
> Hi Eric,
> 
> There is enough disk space in the fs.
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.9G  6.6G  31% /
> /dev/sdb              147G  188M  140G   1% /mnt
> none                  854M     0  854M   0% /dev/shm
> /dev/sdh              5.0G   25M  5.0G   1% /mymountpoint
> 
> Also, the output of modinfo is as follows,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
> filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko
> license:        GPL
> description:    SGI XFS with ACLs, security attributes, large block
> numbers, no debug enabled
> author:         Silicon Graphics, Inc.
> srcversion:     C7114C18263E3067C64F2BC
> depends:
> vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS
                                                        ^^^^^^^^

That's another potential cause of the problem, though blowing a
stack normally results in a crash rather than a shutdown...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: input/output error
  2003-03-07 16:37 input/output error Brent Clements
@ 2003-03-07 17:00 ` Trond Myklebust
  0 siblings, 0 replies; 18+ messages in thread
From: Trond Myklebust @ 2003-03-07 17:00 UTC (permalink / raw)
  To: Brent Clements; +Cc: NFS

>>>>> " " == Brent Clements <bclem@rice.edu> write

     > rtcfiler:/users/users /users nfs
     > rw,v3,rsize=32768,wsize=32768,soft,udp,lock,addr=rtcfiler 0 0
                                     ^^^^

Read up on the 'soft' mount option in 'man 5 nfs'.

Cheers,
  Trond


-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* input/output error
@ 2003-03-07 16:37 Brent Clements
  2003-03-07 17:00 ` Trond Myklebust
  0 siblings, 1 reply; 18+ messages in thread
From: Brent Clements @ 2003-03-07 16:37 UTC (permalink / raw)
  To: NFS

I am getting the following error when I do this.

time dd if=/dev/zero of=/users/testfile bs=16k count=65536

dd: writing `/users/testf2': Input/output error
41986+0 records in
41985+0 records out

real    0m11.301s
user    0m0.040s
sys     0m1.779s

I'm only getting partial writes of the file. IE..it never completes
writing the file. 

I am using an hp/ux fileserver as my nfs server and it is version 3

The following is the output from /proc/mounts on my linux-2.4.20 ia64 
client.

rtcfiler:/users/users /users nfs
rw,v3,rsize=32768,wsize=32768,soft,udp,lock,addr=rtcfiler 0 0


Does anyone have any idea how I can debug why its doing this? Or has
anyone seen this before?

Thanks,
Brent Clements
Rice University




-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Input/output error
@ 2002-09-20 14:06 Kenneth Johansson
  0 siblings, 0 replies; 18+ messages in thread
From: Kenneth Johansson @ 2002-09-20 14:06 UTC (permalink / raw)
  To: Mtd

I got this on a ppc system running 2-4-19-pre8.

Five files is now in a state where they show up in the file system but
can't be accessed or stat (ls works but ls -l do not).

I see that a few fixes has gone in since this version. Could any of them
fix a problem like this?



-- 
Kenneth Johansson	
Ericsson AB                       Tel: +46 8 404 71 83
Borgafjordsgatan 9                Fax: +46 8 404 72 72
164 80 Stockholm                  kenneth.johansson@etx.ericsson.se

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-25 22:50       ` Horst von Brand
@ 2001-11-26  0:05         ` Russell King
  0 siblings, 0 replies; 18+ messages in thread
From: Russell King @ 2001-11-26  0:05 UTC (permalink / raw)
  To: Horst von Brand; +Cc: Tom Eastep, Hartmut Holz, linux-kernel

On Sun, Nov 25, 2001 at 07:50:06PM -0300, Horst von Brand wrote:
> > > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
> > >   Clear? yes
> 
> [Ad nauseam]
> 
> Saw this with 2.4.15pre[89]

This is the FS bug that Al Viro has fixed for 2.4.15/2.5.0.

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html


^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-23 19:10     ` Tom Eastep
@ 2001-11-25 22:50       ` Horst von Brand
  2001-11-26  0:05         ` Russell King
  0 siblings, 1 reply; 18+ messages in thread
From: Horst von Brand @ 2001-11-25 22:50 UTC (permalink / raw)
  To: Tom Eastep; +Cc: Hartmut Holz, linux-kernel

Tom Eastep <teastep@shorewall.net> said:

[...]

> A "me too" here (2.4.15 final) -- ran fsck on the /var partition and saw 
> output similar to that below.
> >
> > e2fsck output:
> > --------------
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005.
> > Clear? yes
> >
> > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
> >   Clear? yes

[Ad nauseam]

Saw this with 2.4.15pre[89]
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-23 18:29     ` Andreas Dilger
  2001-11-23 18:50       ` Marcus Grando
@ 2001-11-23 20:20       ` Hartmut Holz
  1 sibling, 0 replies; 18+ messages in thread
From: Hartmut Holz @ 2001-11-23 20:20 UTC (permalink / raw)
  To: Andreas Dilger; +Cc: linux-kernel

Andreas Dilger wrote:


> 
> I take it that this is after a normal shutdown where you are sure that
> the filesystem was unmounted cleanly?  It looks like a case where these
> files are deleted, but held open by a process.
> 


It's a normal shutdown, every filesystem is unmounted cleanly, no
complains from the os.


> Could you please try the following:
> - "telinit 1" to change into single user mode
> - make sure all of the above processes are stopped (check via ps, and
>   "/etc/rc.d/init.d/foo stop" for each one
> - "lsof | grep /var" to see if any files are still open on /var
> - umount /var
> - e2fsck -f /dev/hdX


There are a few processes running: news, something called minilgd.Even 
with these programs running lsof shows no output. If I kill these 
programs and make a normal shutdown, it loocked much better. Only a 
complain about the keyboard lock.

If I acted exactly like your advise, there is no problem with the restart.

The processes running in 2.4.15 single user mode are exactly the same as 
in 2.4.14.


Regards

Hartmut

 




^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-23 15:43   ` Hartmut Holz
  2001-11-23 18:29     ` Andreas Dilger
@ 2001-11-23 19:10     ` Tom Eastep
  2001-11-25 22:50       ` Horst von Brand
  1 sibling, 1 reply; 18+ messages in thread
From: Tom Eastep @ 2001-11-23 19:10 UTC (permalink / raw)
  To: Hartmut Holz, linux-kernel

On Friday 23 November 2001 07:43 am, Hartmut Holz wrote:
> Rasmus Bøg Hansen wrote:
> > On Thu, 22 Nov 2001, Marcus Grando wrote:
> >>On try start syslog deamon occur this errrors "Input/output error" on
> >> many archives /var directory.
> >
> > Try to run fsck on the /var partition. Also you should check the disk
> > for bad blocks. What output do you get from the kernel ('dmesg',
> > /var/log/messages etc.)?
> >
> > It could be a bad disk developing bad sectors.
> >
> > Rasmus
>
> On my machine (2.4.15 final) it is the same behaviour.  After reboot
> the lock files (and only the lock files) are corrupt. With 2.4.14 and
> 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97

A "me too" here (2.4.15 final) -- ran fsck on the /var partition and saw 
output similar to that below.
>
> e2fsck output:
> --------------
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005.
> Clear? yes
>
> Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
>   Clear? yes
>
> Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007.
> Clear? yes
>
> Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008.
>   Clear? yes
>
> Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009.
>   Clear? yes
>
> Entry 'random' in /lock/subsys (30001) has deleted/unused inode 30012.
> Clear? yes
>
> Entry 'netfs' in /lock/subsys (30001) has deleted/unused inode 30013.
> Clear? yes
>
> Entry 'autofs' in /lock/subsys (30001) has deleted/unused inode 30014.
> Clear? yes
>
> Entry 'local' in /lock/subsys (30001) has deleted/unused inode 30029.
> Clear? yes
>
> Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009.
> Clear? yes
>
> Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010.
> Clear? yes
>
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
>
> /var: ***** FILE SYSTEM WAS MODIFIED *****
> /var: 23017/66000 files (5.9% non-contiguous), 115364/263168 blocks
>

-Tom
-- 
Tom Eastep    \  teastep@shorewall.net
AIM: tmeastep  \  http://www.shorewall.net
ICQ: #60745924  \_________________________

^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-23 18:29     ` Andreas Dilger
@ 2001-11-23 18:50       ` Marcus Grando
  2001-11-23 20:20       ` Hartmut Holz
  1 sibling, 0 replies; 18+ messages in thread
From: Marcus Grando @ 2001-11-23 18:50 UTC (permalink / raw)
  To: Andreas Dilger; +Cc: Hartmut Holz, linux-kernel


	Hello,

	I have this problem too, i execute fsck more 20 times and not 
resolv this problem.

	Execute badblocks(8) and not found bad blocks

	I note before linux-2.4.15-pre8 begin this problems.

	Changelog-pre8:
	 - Andrew Morton: fix ext3/minix/sysv fsync behaviour.

	Maybe? I don´t no.

Regards,

Marcus Grando


On Fri, 23 Nov 2001, Andreas Dilger wrote:

> On Nov 23, 2001  16:43 +0100, Hartmut Holz wrote:
> > On my machine (2.4.15 final) it is the same behaviour.  After reboot
> > the lock files (and only the lock files) are corrupt. With 2.4.14 and 
> > 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97
> > 
> > e2fsck output:
> > --------------
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
> > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
> > Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
> > Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
> > Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
> > Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
> > Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 
> 
> I take it that this is after a normal shutdown where you are sure that
> the filesystem was unmounted cleanly?  It looks like a case where these
> files are deleted, but held open by a process.
> 
> Could you please try the following:
> - "telinit 1" to change into single user mode
> - make sure all of the above processes are stopped (check via ps, and
>   "/etc/rc.d/init.d/foo stop" for each one
> - "lsof | grep /var" to see if any files are still open on /var
> - umount /var
> - e2fsck -f /dev/hdX
> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Marcus Grando
Administrador de Rede
UNIVALI


^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-23 15:43   ` Hartmut Holz
@ 2001-11-23 18:29     ` Andreas Dilger
  2001-11-23 18:50       ` Marcus Grando
  2001-11-23 20:20       ` Hartmut Holz
  2001-11-23 19:10     ` Tom Eastep
  1 sibling, 2 replies; 18+ messages in thread
From: Andreas Dilger @ 2001-11-23 18:29 UTC (permalink / raw)
  To: Hartmut Holz; +Cc: linux-kernel

On Nov 23, 2001  16:43 +0100, Hartmut Holz wrote:
> On my machine (2.4.15 final) it is the same behaviour.  After reboot
> the lock files (and only the lock files) are corrupt. With 2.4.14 and 
> 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97
> 
> e2fsck output:
> --------------
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
> Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
> Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
> Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
> Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
> Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
> Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 

I take it that this is after a normal shutdown where you are sure that
the filesystem was unmounted cleanly?  It looks like a case where these
files are deleted, but held open by a process.

Could you please try the following:
- "telinit 1" to change into single user mode
- make sure all of the above processes are stopped (check via ps, and
  "/etc/rc.d/init.d/foo stop" for each one
- "lsof | grep /var" to see if any files are still open on /var
- umount /var
- e2fsck -f /dev/hdX

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/


^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-22 13:17 ` Rasmus Bøg Hansen
@ 2001-11-23 15:43   ` Hartmut Holz
  2001-11-23 18:29     ` Andreas Dilger
  2001-11-23 19:10     ` Tom Eastep
  0 siblings, 2 replies; 18+ messages in thread
From: Hartmut Holz @ 2001-11-23 15:43 UTC (permalink / raw)
  To: linux-kernel

Rasmus Bøg Hansen wrote:

> On Thu, 22 Nov 2001, Marcus Grando wrote:
> 
> 
>>On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.
>>
> 
> Try to run fsck on the /var partition. Also you should check the disk 
> for bad blocks. What output do you get from the kernel ('dmesg', 
> /var/log/messages etc.)?
> 
> It could be a bad disk developing bad sectors.
> 
> Rasmus
> 
> 

On my machine (2.4.15 final) it is the same behaviour.  After reboot
the lock files (and only the lock files) are corrupt. With 2.4.14 and 
2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97

e2fsck output:
--------------
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
Clear? yes

Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
  Clear? yes

Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
Clear? yes

Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
  Clear? yes

Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
  Clear? yes

Entry 'random' in /lock/subsys (30001) has deleted/unused inode 30012. 
Clear? yes

Entry 'netfs' in /lock/subsys (30001) has deleted/unused inode 30013. 
Clear? yes

Entry 'autofs' in /lock/subsys (30001) has deleted/unused inode 30014. 
Clear? yes

Entry 'local' in /lock/subsys (30001) has deleted/unused inode 30029. 
Clear? yes

Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
Clear? yes

Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 
Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/var: ***** FILE SYSTEM WAS MODIFIED *****
/var: 23017/66000 files (5.9% non-contiguous), 115364/263168 blocks



^ permalink raw reply	[flat|nested] 18+ messages in thread
* Re: Input/output error
  2001-11-22 13:04 Marcus Grando
@ 2001-11-22 13:17 ` Rasmus Bøg Hansen
  2001-11-23 15:43   ` Hartmut Holz
  0 siblings, 1 reply; 18+ messages in thread
From: Rasmus Bøg Hansen @ 2001-11-22 13:17 UTC (permalink / raw)
  To: Marcus Grando; +Cc: linux-kernel

On Thu, 22 Nov 2001, Marcus Grando wrote:

> On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.

Try to run fsck on the /var partition. Also you should check the disk 
for bad blocks. What output do you get from the kernel ('dmesg', 
/var/log/messages etc.)?

It could be a bad disk developing bad sectors.

Rasmus

-- 
-- [ Rasmus 'Møffe' Bøg Hansen ] ---------------------------------------
DISCLAIMER: Microsoft, Windows, Windows 98, Bugs, Lacking features, IRQ
conflicts, System crashes, Non-functional multitasking, the Y2K problem
and the Blue Screen of Death are registered trademarks of
Microsoft, Corp., Redmond, USA.
--------------------------------- [ moffe at amagerkollegiet dot dk ] --


^ permalink raw reply	[flat|nested] 18+ messages in thread
* Input/output error
@ 2001-11-22 13:04 Marcus Grando
  2001-11-22 13:17 ` Rasmus Bøg Hansen
  0 siblings, 1 reply; 18+ messages in thread
From: Marcus Grando @ 2001-11-22 13:04 UTC (permalink / raw)
  To: linux-kernel

Hi list,

After reboot in kernel 2.4.15-pre9 this problem occur:

On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.

I mount the /var at boot linux single, and when try ls on /var/run list all archives .pid, 
before try ls -la on /var/run, not list all archives.
Try remove .pid archives and the error occur again.

/var partition use ext2

Any suggestion?

Tanks in advance,

Regards
Marcus Grando

	


^ permalink raw reply	[flat|nested] 18+ messages in thread
end of thread, other threads:[~2011-02-23 21:52 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-02-22 15:44 Input/Output error Srinivasan T
2011-02-22 20:21 ` Eric Sandeen
2011-02-23  6:33   ` Srinivasan T
2011-02-23 16:32     ` Eric Sandeen
2011-02-23 21:55     ` Dave Chinner
2011-02-23  0:47 ` Dave Chinner
  -- strict thread matches above, loose matches on Subject: below --
2003-03-07 16:37 input/output error Brent Clements
2003-03-07 17:00 ` Trond Myklebust
2002-09-20 14:06 Input/output error Kenneth Johansson
2001-11-22 13:04 Marcus Grando
2001-11-22 13:17 ` Rasmus Bøg Hansen
2001-11-23 15:43   ` Hartmut Holz
2001-11-23 18:29     ` Andreas Dilger
2001-11-23 18:50       ` Marcus Grando
2001-11-23 20:20       ` Hartmut Holz
2001-11-23 19:10     ` Tom Eastep
2001-11-25 22:50       ` Horst von Brand
2001-11-26  0:05         ` Russell King

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.