Stale NFS file handle - persistent error

Stale NFS file handle - persistent error

[Carlos Jiménez]

Hello everybody,

I have a computer acting as an NFS Server running a Debian based distro 
and several hosts running CentOS 6.2 acting as NFS clients (for a 
virtualization environment). Every time more often "Stale NFS file 
handle" errors appear when trying to copy a file from one directory to 
another (both in the nfs shared directory).
In the server side there are no related information in the logs, but 
this is the output of one of the nfs clients:

Wed Aug 29 11:52:57 2012 [TM][E]: clone: Command "cd 
/var/lib/one/datastores/0/32; cp -r 
../../1/5c0455eb494fd43b5c7c576e0c642fbe 
/var/lib/one/datastores/0/32/disk.0" failed: cp: reading 
`../../1/5c0455eb494fd43b5c7c576e0c642fbe': Stale NFS file handle

I've checked that path and filenames are correct. The file itself is a 
300MG sized and the error appears aprox. 30 minutes after the start of 
the copy, so I guess that it doesn't finish the file copy.
I usually tried unmounting and remounting the shared directory. In fact, 
initially it was enough and it was possible to fully copy the file 
(after aprox. 30 minutes), moreover because the error took place from 
time to time. But now, the problem persists after doing it (even after 
rebooting the clients).
Additionally, I've tried doing the same operation (to copy the same file 
from a directory to another) locally in the NFS Server and it takes just 
a minute (aprox.).

Could you give me advice on troubleshooting it?


Thanks in advance.

Carlos.

Re: Stale NFS file handle - persistent error

[J. Bruce Fields]

On Wed, Aug 29, 2012 at 03:06:07PM +0200, Carlos Jiménez wrote:
> I have a computer acting as an NFS Server running a Debian based
> distro

Could you be more specific?  What kernel version exactly, and what
filesystem are you exporting, and what are the export options?

What version of NFS are you using?

> and several hosts running CentOS 6.2 acting as NFS clients
> (for a virtualization environment). Every time more often "Stale NFS
> file handle" errors appear when trying to copy a file from one
> directory to another (both in the nfs shared directory).

I assume you're not doing anything else with either file or directory at
the same time?  (E.g., not deleting or renaming the file while it's
being copied?)

--b.

> In the server side there are no related information in the logs, but
> this is the output of one of the nfs clients:
> 
> Wed Aug 29 11:52:57 2012 [TM][E]: clone: Command "cd
> /var/lib/one/datastores/0/32; cp -r
> ../../1/5c0455eb494fd43b5c7c576e0c642fbe
> /var/lib/one/datastores/0/32/disk.0" failed: cp: reading
> `../../1/5c0455eb494fd43b5c7c576e0c642fbe': Stale NFS file handle
> 
> I've checked that path and filenames are correct. The file itself is
> a 300MG sized and the error appears aprox. 30 minutes after the
> start of the copy, so I guess that it doesn't finish the file copy.
> I usually tried unmounting and remounting the shared directory. In
> fact, initially it was enough and it was possible to fully copy the
> file (after aprox. 30 minutes), moreover because the error took
> place from time to time. But now, the problem persists after doing
> it (even after rebooting the clients).
> Additionally, I've tried doing the same operation (to copy the same
> file from a directory to another) locally in the NFS Server and it
> takes just a minute (aprox.).
> 
> Could you give me advice on troubleshooting it?
> 
> 
> Thanks in advance.
> 
> Carlos.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Stale NFS file handle - persistent error

[Carlos Jiménez]

Hi,

Thank you for your reply.
> Could you be more specific?  What kernel version exactly, and what
> filesystem are you exporting, and what are the export options?
Kernel version: 2.4.27-3-386
Filesystem exported: ext3
Export options: rw,no_root_squash,sync

I've shared/exported to the full network 192.168.180.0/24.
The exported directory is /opt/virtual/one and in /opt/virtual it is 
mounted /dev/cciss/c0d0p4 (with default options), which is a 256GB (8GB 
used) partition over a 3 disks RAID5.

>
> What version of NFS are you using?
>
Vers 3

> I assume you're not doing anything else with either file or directory at
> the same time?  (E.g., not deleting or renaming the file while it's
> being copied?)
>
>
I'm not doing anything else with either file or directory at the same time.

Re: Stale NFS file handle - persistent error

[Carlos Jiménez]

Also, I've tried copying a file at a client machine from the NFS Server 
(to /dev/null) and it outputs the same stale error:

[root@host1 1]# cp -v 5c0455eb494fd43b5c7c576e0c642fbe /dev/null
cp: overwrite `/dev/null'? y
`5c0455eb494fd43b5c7c576e0c642fbe' -> `/dev/null'
cp: reading `5c0455eb494fd43b5c7c576e0c642fbe': Stale NFS file handle


Cheers,

Carlos.

On 08/30/2012 08:45 AM, Carlos Jiménez wrote:
> Hi,
>
> Thank you for your reply.
>> Could you be more specific?  What kernel version exactly, and what
>> filesystem are you exporting, and what are the export options?
> Kernel version: 2.4.27-3-386
> Filesystem exported: ext3
> Export options: rw,no_root_squash,sync
>
> I've shared/exported to the full network 192.168.180.0/24.
> The exported directory is /opt/virtual/one and in /opt/virtual it is 
> mounted /dev/cciss/c0d0p4 (with default options), which is a 256GB 
> (8GB used) partition over a 3 disks RAID5.
>
>>
>> What version of NFS are you using?
>>
> Vers 3
>
>> I assume you're not doing anything else with either file or directory at
>> the same time?  (E.g., not deleting or renaming the file while it's
>> being copied?)
>>
>>
> I'm not doing anything else with either file or directory at the same 
> time.
>

Re: Stale NFS file handle - persistent error

[Bryan Schumaker]

On 08/30/2012 02:45 AM, Carlos Jiménez wrote:
> Hi,
> 
> Thank you for your reply.
>> Could you be more specific?  What kernel version exactly, and what
>> filesystem are you exporting, and what are the export options?
> Kernel version: 2.4.27-3-386

2.4.27 is incredibly old, do you have access to the server and can you try a more recent kernel?

- Bryan

> Filesystem exported: ext3
> Export options: rw,no_root_squash,sync
> 
> I've shared/exported to the full network 192.168.180.0/24.
> The exported directory is /opt/virtual/one and in /opt/virtual it is mounted /dev/cciss/c0d0p4 (with default options), which is a 256GB (8GB used) partition over a 3 disks RAID5.
> 
>>
>> What version of NFS are you using?
>>
> Vers 3
> 
>> I assume you're not doing anything else with either file or directory at
>> the same time?  (E.g., not deleting or renaming the file while it's
>> being copied?)
>>
>>
> I'm not doing anything else with either file or directory at the same time.
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Stale NFS file handle - persistent error

[J. Bruce Fields]

On Thu, Aug 30, 2012 at 12:02:04PM -0400, Bryan Schumaker wrote:
> On 08/30/2012 02:45 AM, Carlos Jiménez wrote:
> > Hi,
> > 
> > Thank you for your reply.
> >> Could you be more specific?  What kernel version exactly, and what
> >> filesystem are you exporting, and what are the export options?
> > Kernel version: 2.4.27-3-386
> 
> 2.4.27 is incredibly old, do you have access to the server and can you try a more recent kernel?

Yes, we can't make much use of 2.4 bug reports at this point; you should
probably report this to your distro, or try a more recent kernel.

--b.

> 
> - Bryan
> 
> > Filesystem exported: ext3
> > Export options: rw,no_root_squash,sync
> > 
> > I've shared/exported to the full network 192.168.180.0/24.
> > The exported directory is /opt/virtual/one and in /opt/virtual it is mounted /dev/cciss/c0d0p4 (with default options), which is a 256GB (8GB used) partition over a 3 disks RAID5.
> > 
> >>
> >> What version of NFS are you using?
> >>
> > Vers 3
> > 
> >> I assume you're not doing anything else with either file or directory at
> >> the same time?  (E.g., not deleting or renaming the file while it's
> >> being copied?)
> >>
> >>
> > I'm not doing anything else with either file or directory at the same time.
> > 
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Stale NFS file handle - persistent error

[Carlos Jiménez]

Ok, you are right.
Certainly, it is a very old kernel because we had problems with the SCSI 
controller in new distros with 2.6-3 kernel. I think there is another 
computer available, so I think the best choice will be to try to move to 
that one.

Thanks guys for your fast replies and your time.

Carlos.

On 08/30/2012 08:59 PM, J. Bruce Fields wrote:
> On Thu, Aug 30, 2012 at 12:02:04PM -0400, Bryan Schumaker wrote:
>> On 08/30/2012 02:45 AM, Carlos Jiménez wrote:
>>> Hi,
>>>
>>> Thank you for your reply.
>>>> Could you be more specific?  What kernel version exactly, and what
>>>> filesystem are you exporting, and what are the export options?
>>> Kernel version: 2.4.27-3-386
>> 2.4.27 is incredibly old, do you have access to the server and can you try a more recent kernel?
> Yes, we can't make much use of 2.4 bug reports at this point; you should
> probably report this to your distro, or try a more recent kernel.
>
> --b.
>
>> - Bryan
>>
>>> Filesystem exported: ext3
>>> Export options: rw,no_root_squash,sync
>>>
>>> I've shared/exported to the full network 192.168.180.0/24.
>>> The exported directory is /opt/virtual/one and in /opt/virtual it is mounted /dev/cciss/c0d0p4 (with default options), which is a 256GB (8GB used) partition over a 3 disks RAID5.
>>>
>>>> What version of NFS are you using?
>>>>
>>> Vers 3
>>>
>>>> I assume you're not doing anything else with either file or directory at
>>>> the same time?  (E.g., not deleting or renaming the file while it's
>>>> being copied?)
>>>>
>>>>
>>> I'm not doing anything else with either file or directory at the same time.
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html