Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Ashish Sangwan]

Hi,

Our NFS filer can sometimes return same inode number for different directories.
For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
and dir4 might end up returning the same inode number to the client.
Though it can never happen that inode numbers will be same for two
directories and also there parent is same. Can linux client handle
this case? What issues it can cause?
https://lkml.org/lkml/2006/10/2/346
I stumbled upon this thread where it is written that nfs client can
handle this but userspace will see inode collisions. Given that this
will happen only for directories, userspace utils logic might not get
affected from this as hardlinks on directories are not possible. But
the thread is really old. Wanted to confirm if this holds true even
now.

Thanks,
Ashish

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

On Thu, Dec 13 2018, Ashish Sangwan wrote:

> Hi,
>
> Our NFS filer can sometimes return same inode number for different directories.

Why?

> For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
> and dir4 might end up returning the same inode number to the client.
> Though it can never happen that inode numbers will be same for two
> directories and also there parent is same. Can linux client handle
> this case? What issues it can cause?

As long as the file handles are different, the Linux client won't really
notice.
Problems might occur with applications which check inode numbers.
I don't know of any that would be confused by directories having the
same inode number, but that doesn't mean there aren't any.

> https://lkml.org/lkml/2006/10/2/346

This is ancient!  It is mostly about the NFS server, not the client.
Filesystems that NFSd is exporting need to be careful to provide unique
file handles.

> I stumbled upon this thread where it is written that nfs client can
> handle this but userspace will see inode collisions. Given that this
> will happen only for directories, userspace utils logic might not get
> affected from this as hardlinks on directories are not possible. But
> the thread is really old. Wanted to confirm if this holds true even
> now.

I don't think anything important has changed.  The server must return
unquie filehandles.  It should return unique inode numbers.  User-space
may or may not get confused if it doesn't.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Ashish Sangwan]

Thanks for the clarification!

On Thu, 13 Dec 2018, 4:15 am NeilBrown <neilb@suse.com wrote:
>
> On Thu, Dec 13 2018, Ashish Sangwan wrote:
>
> > Hi,
> >
> > Our NFS filer can sometimes return same inode number for different directories.
>
> Why?
The NFS fileserver is handling file systems over different nodes at
the same time.
To the client however, we want to present with a single pseudo fsid
(sent as part of the getattr) so that submounts can be avoided.
We have assigned unique numbers to identify different file systems and
we append those numbers to the actual on-disk inode numbers to avoid
the collision.  But in some rare cases there is not enough free bits
in the inode to accommodate the unique filesystem identifier.

>
> > For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
> > and dir4 might end up returning the same inode number to the client.
> > Though it can never happen that inode numbers will be same for two
> > directories and also there parent is same. Can linux client handle
> > this case? What issues it can cause?
>
> As long as the file handles are different, the Linux client won't really
> notice.
A naive question here. This should also not cause any issue in the VFS layer?

> Problems might occur with applications which check inode numbers.
> I don't know of any that would be confused by directories having the
> same inode number, but that doesn't mean there aren't any.
>
> > https://lkml.org/lkml/2006/10/2/346
>
> This is ancient!  It is mostly about the NFS server, not the client.
> Filesystems that NFSd is exporting need to be careful to provide unique
> file handles.
>
> > I stumbled upon this thread where it is written that nfs client can
> > handle this but userspace will see inode collisions. Given that this
> > will happen only for directories, userspace utils logic might not get
> > affected from this as hardlinks on directories are not possible. But
> > the thread is really old. Wanted to confirm if this holds true even
> > now.
>
> I don't think anything important has changed.  The server must return
> unquie filehandles.  It should return unique inode numbers.  User-space
> may or may not get confused if it doesn't.
Understood that this has to be fixed ultimately. Just wanted to have
an idea regarding the severity of the issue.
>
> NeilBrown

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

On Thu, Dec 13 2018, Ashish Sangwan wrote:

> Thanks for the clarification!
>
> On Thu, 13 Dec 2018, 4:15 am NeilBrown <neilb@suse.com wrote:
>>
>> On Thu, Dec 13 2018, Ashish Sangwan wrote:
>>
>> > Hi,
>> >
>> > Our NFS filer can sometimes return same inode number for different directories.
>>
>> Why?
> The NFS fileserver is handling file systems over different nodes at
> the same time.
> To the client however, we want to present with a single pseudo fsid
> (sent as part of the getattr) so that submounts can be avoided.
> We have assigned unique numbers to identify different file systems and
> we append those numbers to the actual on-disk inode numbers to avoid
> the collision.  But in some rare cases there is not enough free bits
> in the inode to accommodate the unique filesystem identifier.
>
>>
>> > For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
>> > and dir4 might end up returning the same inode number to the client.
>> > Though it can never happen that inode numbers will be same for two
>> > directories and also there parent is same. Can linux client handle
>> > this case? What issues it can cause?
>>
>> As long as the file handles are different, the Linux client won't really
>> notice.
> A naive question here. This should also not cause any issue in the VFS layer?

No, the VFS layer won't notice duplicates.

NeilBrown


>
>> Problems might occur with applications which check inode numbers.
>> I don't know of any that would be confused by directories having the
>> same inode number, but that doesn't mean there aren't any.
>>
>> > https://lkml.org/lkml/2006/10/2/346
>>
>> This is ancient!  It is mostly about the NFS server, not the client.
>> Filesystems that NFSd is exporting need to be careful to provide unique
>> file handles.
>>
>> > I stumbled upon this thread where it is written that nfs client can
>> > handle this but userspace will see inode collisions. Given that this
>> > will happen only for directories, userspace utils logic might not get
>> > affected from this as hardlinks on directories are not possible. But
>> > the thread is really old. Wanted to confirm if this holds true even
>> > now.
>>
>> I don't think anything important has changed.  The server must return
>> unquie filehandles.  It should return unique inode numbers.  User-space
>> may or may not get confused if it doesn't.
> Understood that this has to be fixed ultimately. Just wanted to have
> an idea regarding the severity of the issue.
>>
>> NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[J. Bruce Fields]

On Thu, Dec 13, 2018 at 09:17:31AM +0530, Ashish Sangwan wrote:
> Thanks for the clarification!
> 
> On Thu, 13 Dec 2018, 4:15 am NeilBrown <neilb@suse.com wrote:
> >
> > On Thu, Dec 13 2018, Ashish Sangwan wrote:
> >
> > > Hi,
> > >
> > > Our NFS filer can sometimes return same inode number for different directories.
> >
> > Why?
> The NFS fileserver is handling file systems over different nodes at
> the same time.
> To the client however, we want to present with a single pseudo fsid
> (sent as part of the getattr) so that submounts can be avoided.

Out of curiosity, why do you want to avoid submounts?

--b.

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Ashish Sangwan]

On Fri, Jan 11, 2019 at 2:03 AM J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Thu, Dec 13, 2018 at 09:17:31AM +0530, Ashish Sangwan wrote:
> > Thanks for the clarification!
> >
> > On Thu, 13 Dec 2018, 4:15 am NeilBrown <neilb@suse.com wrote:
> > >
> > > On Thu, Dec 13 2018, Ashish Sangwan wrote:
> > >
> > > > Hi,
> > > >
> > > > Our NFS filer can sometimes return same inode number for different directories.
> > >
> > > Why?
> > The NFS fileserver is handling file systems over different nodes at
> > the same time.
> > To the client however, we want to present with a single pseudo fsid
> > (sent as part of the getattr) so that submounts can be avoided.
>
> Out of curiosity, why do you want to avoid submounts?

Apologies for late reply. Didn't notice I have a new mail for this thread.
Some of the user space utils doesn't work well on mount points.
For example, "rm -rf" on the mount point doesn't work. (unlink fails with EBUSY)
GNU coreutils "mv" also doesn't work (rename fails with EBUSY)
(all-though even if sub-mounts are avoided, we give back EXDEV as part
of rename but that is internally handled by mv which performs
write/unlink in that case)
Every time user have to perform such operations, we have to tell them
to explicitly perform a umount first.
>
> --b.

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Frank Filz]

On 12/12/18 10:57 AM, Ashish Sangwan wrote:
> Hi,
>
> Our NFS filer can sometimes return same inode number for different directories.
> For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
> and dir4 might end up returning the same inode number to the client.
> Though it can never happen that inode numbers will be same for two
> directories and also there parent is same. Can linux client handle
> this case? What issues it can cause?
> https://lkml.org/lkml/2006/10/2/346
> I stumbled upon this thread where it is written that nfs client can
> handle this but userspace will see inode collisions. Given that this
> will happen only for directories, userspace utils logic might not get
> affected from this as hardlinks on directories are not possible. But
> the thread is really old. Wanted to confirm if this holds true even
> now.
>
> Thanks,
> Ashish

The find tool will detect the collisions and report them, and if I 
recall it stops proceeding down the directory tree from the duplicate 
inode number. I know for sure I've seen it complain, and pretty sure I 
saw it stop descending the tree into the directory with the duplicate 
inode number.


Frank

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Ashish Sangwan]

On Thu, Dec 13, 2018 at 9:56 PM Frank Filz <ffilz@redhat.com> wrote:
>
> On 12/12/18 10:57 AM, Ashish Sangwan wrote:
> > Hi,
> >
> > Our NFS filer can sometimes return same inode number for different directories.
> > For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
> > and dir4 might end up returning the same inode number to the client.
> > Though it can never happen that inode numbers will be same for two
> > directories and also there parent is same. Can linux client handle
> > this case? What issues it can cause?
> > https://lkml.org/lkml/2006/10/2/346
> > I stumbled upon this thread where it is written that nfs client can
> > handle this but userspace will see inode collisions. Given that this
> > will happen only for directories, userspace utils logic might not get
> > affected from this as hardlinks on directories are not possible. But
> > the thread is really old. Wanted to confirm if this holds true even
> > now.
> >
> > Thanks,
> > Ashish
>
> The find tool will detect the collisions and report them, and if I
> recall it stops proceeding down the directory tree from the duplicate
> inode number. I know for sure I've seen it complain, and pretty sure I
> saw it stop descending the tree into the directory with the duplicate
> inode number.
>

True. Was just looking into the find code to check how they detect cycles.
They are using pair of "device node : inode number" so find will
certainly complain.

>
> Frank
>

Re: Handling of duplicate inode numbers for the directories in the nfs v3 kernel client

[Jeff Layton]

On Fri, 2018-12-14 at 10:41 +0530, Ashish Sangwan wrote:
> On Thu, Dec 13, 2018 at 9:56 PM Frank Filz <ffilz@redhat.com> wrote:
> > On 12/12/18 10:57 AM, Ashish Sangwan wrote:
> > > Hi,
> > > 
> > > Our NFS filer can sometimes return same inode number for different directories.
> > > For example /mnt/dir1/dir2 and /mnt/dir3/dir4, in same rare cases dir2
> > > and dir4 might end up returning the same inode number to the client.
> > > Though it can never happen that inode numbers will be same for two
> > > directories and also there parent is same. Can linux client handle
> > > this case? What issues it can cause?
> > > https://lkml.org/lkml/2006/10/2/346
> > > I stumbled upon this thread where it is written that nfs client can
> > > handle this but userspace will see inode collisions. Given that this
> > > will happen only for directories, userspace utils logic might not get
> > > affected from this as hardlinks on directories are not possible. But
> > > the thread is really old. Wanted to confirm if this holds true even
> > > now.
> > > 
> > > Thanks,
> > > Ashish
> > 
> > The find tool will detect the collisions and report them, and if I
> > recall it stops proceeding down the directory tree from the duplicate
> > inode number. I know for sure I've seen it complain, and pretty sure I
> > saw it stop descending the tree into the directory with the duplicate
> > inode number.
> > 
> 
> True. Was just looking into the find code to check how they detect cycles.
> They are using pair of "device node : inode number" so find will
> certainly complain.
> 

You may also experience problems with some archiving tools -- e.g. tar,
cpio or rsync. Many of those programs rely on looking at inode numbers
to detect hardlinks.

Neil is quite right that the kernel NFS client generally doesn't care
about the inode number though.

-- 
Jeff Layton <jlayton@redhat.com>