Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Andrew Walker]

On my Windows server I mounted multiple NTFS volumes within the same
share and played around until I was able to create directories with
the same fileid number.

 ```
PS C:\SHARE> fsutil file queryFileID SUBDS/foo
File ID is 0x00000000000000000001000000000026
PS C:\SHARE> fsutil file queryFileID .\SUBDS3\TESTFOLDER\
File ID is 0x00000000000000000001000000000026
```
The Linux SMB client then grants these two directories the same devid
/ inode pair.

```
root@truenas[/tmpcifs]# stat SUBDS/foo
  File: SUBDS/foo
  Size: 0         Blocks: 0          IO Block: 1048576 directory
Device: 5eh/94d Inode: 281474976710694  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-01 13:13:42.425159200 -0800
Modify: 2023-03-01 13:13:42.425159200 -0800
Change: 2023-03-01 13:13:42.425159200 -0800
 Birth: 2023-02-22 10:49:37.331907600 -0800

root@truenas[/tmpcifs]# stat SUBDS3/TESTFOLDER
  File: SUBDS3/TESTFOLDER
  Size: 0         Blocks: 0          IO Block: 1048576 directory
Device: 5eh/94d Inode: 281474976710694  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-01 12:25:56.283589300 -0800
Modify: 2023-03-01 12:25:56.268165700 -0800
Change: 2023-03-01 12:25:56.268165700 -0800
 Birth: 2023-03-01 11:48:09.617084100 -0800
```

Windows identifies mounted volumes via reparse point:
https://learn.microsoft.com/en-us/windows/win32/fileio/determining-whether-a-directory-is-a-volume-mount-point

This is also clearly visible in packet captures.

Andrew

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Paulo Alcantara]

Andrew Walker <awalker@ixsystems.com> writes:

> On my Windows server I mounted multiple NTFS volumes within the same
> share and played around until I was able to create directories with
> the same fileid number.

Did you try it with 'noserverino' mount option?  For more information,
see mount.cifs(8).

Did it work with older kernels?

> Windows identifies mounted volumes via reparse point:
> https://learn.microsoft.com/en-us/windows/win32/fileio/determining-whether-a-directory-is-a-volume-mount-point

Perhaps we could conditionally stop trusting file ids sent by the server
as we currently do for hardlinks when we see these reparse points as
well.

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Andrew Walker]

On Wed, Mar 1, 2023 at 5:37 PM Paulo Alcantara <pc@manguebit.com> wrote:
>
> Andrew Walker <awalker@ixsystems.com> writes:
>
> > On my Windows server I mounted multiple NTFS volumes within the same
> > share and played around until I was able to create directories with
> > the same fileid number.
>
> Did you try it with 'noserverino' mount option?  For more information,
> see mount.cifs(8).

Yes. In this case I get a unique inode number

> Did it work with older kernels?
I only tested with 5.15 kernel. Was there a different algorithm in
older kernels that's worth testing?

> Perhaps we could conditionally stop trusting file ids sent by the server
> as we currently do for hardlinks when we see these reparse points as
> well.

Maybe fail chdir with EXDEV unless mount is with noserverino?

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Steve French]

I would expect when the inode collision is noted that
"cifs_autodisable_serverino()" will get called in the Linux client and
you should see: "Autodisabling the user of server inode numbers on
..."
"Consider mounting with noserverino to silence this message"

If this is easy to setup we could try some tricks when we encounter
such a reparse point - do you have patch ideas for the client to fake
up inode numbers in this case?

On Wed, Mar 1, 2023 at 7:39 PM Andrew Walker <awalker@ixsystems.com> wrote:
>
> On Wed, Mar 1, 2023 at 5:37 PM Paulo Alcantara <pc@manguebit.com> wrote:
> >
> > Andrew Walker <awalker@ixsystems.com> writes:
> >
> > > On my Windows server I mounted multiple NTFS volumes within the same
> > > share and played around until I was able to create directories with
> > > the same fileid number.
> >
> > Did you try it with 'noserverino' mount option?  For more information,
> > see mount.cifs(8).
>
> Yes. In this case I get a unique inode number
>
> > Did it work with older kernels?
> I only tested with 5.15 kernel. Was there a different algorithm in
> older kernels that's worth testing?
>
> > Perhaps we could conditionally stop trusting file ids sent by the server
> > as we currently do for hardlinks when we see these reparse points as
> > well.
>
> Maybe fail chdir with EXDEV unless mount is with noserverino?



-- 
Thanks,

Steve

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Andrew Walker]

On Wed, Mar 1, 2023 at 8:49 PM Steve French <smfrench@gmail.com> wrote:
>
> I would expect when the inode collision is noted that
> "cifs_autodisable_serverino()" will get called in the Linux client and
> you should see: "Autodisabling the user of server inode numbers on
> ..."
> "Consider mounting with noserverino to silence this message"
>
> If this is easy to setup we could try some tricks when we encounter
> such a reparse point - do you have patch ideas for the client to fake
> up inode numbers in this case?

On the Windows server side it's trivial to set up. Just create /
format an NTFS volume (for example new virtual hdd) and add an
additional location (Computer Management -> Storage -> Disk Managment,
right-click on drive and click "Change Drive Letter and Paths"). This
will allow mounting the volume within an existing SMB share (like a
bind mount in Linux). If you don't want to go through setup and just
want to glance at pcaps of Windows Client <-> Windows Server in this
case I have some I can send to interested parties. I don't have patch
ideas at the moment.


Andrew

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Tom Talpey]

On 3/1/2023 8:49 PM, Steve French wrote:
> I would expect when the inode collision is noted that
> "cifs_autodisable_serverino()" will get called in the Linux client and
> you should see: "Autodisabling the user of server inode numbers on
> ..."
> "Consider mounting with noserverino to silence this message"

Why isn't this behavior simply the default? It's going to be
data corruption (sev 1 issue) if the inode number is the same
for two different fileid's, so this seems entirely backwards.

Also, the words "to silence this message" really don't convey
the severity of the situation.

Tom.

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Steve French]

> Why isn't this behavior simply the default?

Without persisted inode numbers (UniqueId) it would cause problems
with hardlinks (ie mounting with noserverino).  We could try a trick
of hashing them with the volume id if we could detect the transition
to a different volume (as original thread was discussing) -
fortunately in Linux you have to walk a path component by component so
might be possible to spot these more easily.

On Thu, Mar 2, 2023 at 1:19 PM Tom Talpey <tom@talpey.com> wrote:
>
> On 3/1/2023 8:49 PM, Steve French wrote:
> > I would expect when the inode collision is noted that
> > "cifs_autodisable_serverino()" will get called in the Linux client and
> > you should see: "Autodisabling the user of server inode numbers on
> > ..."
> > "Consider mounting with noserverino to silence this message"
>
> Why isn't this behavior simply the default? It's going to be
> data corruption (sev 1 issue) if the inode number is the same
> for two different fileid's, so this seems entirely backwards.
>
> Also, the words "to silence this message" really don't convey
> the severity of the situation.
>
> Tom.



-- 
Thanks,

Steve

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Tom Talpey]

On 3/2/2023 2:23 PM, Steve French wrote:
>> Why isn't this behavior simply the default?
> 
> Without persisted inode numbers (UniqueId) it would cause problems
> with hardlinks (ie mounting with noserverino).  We could try a trick
> of hashing them with the volume id if we could detect the transition
> to a different volume (as original thread was discussing) -
> fortunately in Linux you have to walk a path component by component so
> might be possible to spot these more easily.

Well yeah, it can't be a random assignment, and the fileid is only
unique within the scope of a volumeid. Blindly using the server's
fileid as a client inode without checking for a volume crossing is
a client protocol violation, right?


> On Thu, Mar 2, 2023 at 1:19 PM Tom Talpey <tom@talpey.com> wrote:
>>
>> On 3/1/2023 8:49 PM, Steve French wrote:
>>> I would expect when the inode collision is noted that
>>> "cifs_autodisable_serverino()" will get called in the Linux client and
>>> you should see: "Autodisabling the user of server inode numbers on
>>> ..."
>>> "Consider mounting with noserverino to silence this message"
>>
>> Why isn't this behavior simply the default? It's going to be
>> data corruption (sev 1 issue) if the inode number is the same
>> for two different fileid's, so this seems entirely backwards.
>>
>> Also, the words "to silence this message" really don't convey
>> the severity of the situation.
>>
>> Tom.
> 
> 
> 

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Andrew Walker]

On Thu, Mar 2, 2023 at 1:32 PM Tom Talpey <tom@talpey.com> wrote:
>
> On 3/2/2023 2:23 PM, Steve French wrote:
> >> Why isn't this behavior simply the default?
> >
> > Without persisted inode numbers (UniqueId) it would cause problems
> > with hardlinks (ie mounting with noserverino).  We could try a trick
> > of hashing them with the volume id if we could detect the transition
> > to a different volume (as original thread was discussing) -
> > fortunately in Linux you have to walk a path component by component so
> > might be possible to spot these more easily.
>
> Well yeah, it can't be a random assignment, and the fileid is only
> unique within the scope of a volumeid. Blindly using the server's
> fileid as a client inode without checking for a volume crossing is
> a client protocol violation, right?
>
>
> > On Thu, Mar 2, 2023 at 1:19 PM Tom Talpey <tom@talpey.com> wrote:
> >>
> >> On 3/1/2023 8:49 PM, Steve French wrote:
> >>> I would expect when the inode collision is noted that
> >>> "cifs_autodisable_serverino()" will get called in the Linux client and
> >>> you should see: "Autodisabling the user of server inode numbers on
> >>> ..."
> >>> "Consider mounting with noserverino to silence this message"
> >>
> >> Why isn't this behavior simply the default? It's going to be
> >> data corruption (sev 1 issue) if the inode number is the same
> >> for two different fileid's, so this seems entirely backwards.
> >>
> >> Also, the words "to silence this message" really don't convey
> >> the severity of the situation.
> >>
> >> Tom.
> >
> >
> >

Just glancing briefly at the kernel NFS client, it appears there is
dynamic handling for crossing mountpoints:
```
/*
 * nfs_d_automount - Handle crossing a mountpoint on the server
 * @path - The mountpoint
 *
 * When we encounter a mountpoint on the server, we want to set up
 * a mountpoint on the client too, to prevent inode numbers from
 * colliding, and to allow "df" to work properly.
 * On NFSv4, we also want to allow for the fact that different
 * filesystems may be migrated to different servers in a failover
 * situation, and that different filesystems may want to use
 * different security flavours.
 */
struct vfsmount *nfs_d_automount(struct path *path)
{
```
c.f. fs/nfs/namespace.c

and

```
                } else if (S_ISDIR(inode->i_mode)) {
                        inode->i_op =
NFS_SB(sb)->nfs_client->rpc_ops->dir_inode_ops;
                        inode->i_fop = &nfs_dir_operations;
                        inode->i_data.a_ops = &nfs_dir_aops;
                        nfs_inode_init_dir(nfsi);
                        /* Deal with crossing mountpoints */
                        if (fattr->valid & NFS_ATTR_FATTR_MOUNTPOINT ||
                                        fattr->valid &
NFS_ATTR_FATTR_V4_REFERRAL) {
                                if (fattr->valid & NFS_ATTR_FATTR_V4_REFERRAL)
                                        inode->i_op =
&nfs_referral_inode_operations;
                                else
                                        inode->i_op =
&nfs_mountpoint_inode_operations;
                                inode->i_fop = NULL;
                                inode->i_flags |= S_AUTOMOUNT;
                        }
```

in fs/nfs/inode.c

Andrew

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[ronnie sahlberg]

On Fri, 3 Mar 2023 at 05:38, Steve French <smfrench@gmail.com> wrote:
>
> > Why isn't this behavior simply the default?
>
> Without persisted inode numbers (UniqueId) it would cause problems
> with hardlinks (ie mounting with noserverino).  We could try a trick
> of hashing them with the volume id if we could detect the transition
> to a different volume (as original thread was discussing) -
> fortunately in Linux you have to walk a path component by component so
> might be possible to spot these more easily.

Just hashing it with something does not make the problem go away, it
would just make it
more unpredictable to trigger or reproduce  but the very serious issue
with data loss that Tom ponted out still remains.

Maybe a solution is to NOT traverse across these volume transitions at
all in the client and maybe users should be forced to
explicitely mount these different volumes clientside, exactly like how
NFSv3 does and requires clients to deal with this situation.

But yes, it is an extremely serious bug that will cause data loss when
it triggers for files we write to.


>
> On Thu, Mar 2, 2023 at 1:19 PM Tom Talpey <tom@talpey.com> wrote:
> >
> > On 3/1/2023 8:49 PM, Steve French wrote:
> > > I would expect when the inode collision is noted that
> > > "cifs_autodisable_serverino()" will get called in the Linux client and
> > > you should see: "Autodisabling the user of server inode numbers on
> > > ..."
> > > "Consider mounting with noserverino to silence this message"
> >
> > Why isn't this behavior simply the default? It's going to be
> > data corruption (sev 1 issue) if the inode number is the same
> > for two different fileid's, so this seems entirely backwards.
> >
> > Also, the words "to silence this message" really don't convey
> > the severity of the situation.
> >
> > Tom.
>
>
>
> --
> Thanks,
>
> Steve

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Paulo Alcantara]

Andrew Walker <awalker@ixsystems.com> writes:

> On Wed, Mar 1, 2023 at 5:37 PM Paulo Alcantara <pc@manguebit.com> wrote:
>> Did it work with older kernels?
> I only tested with 5.15 kernel. Was there a different algorithm in
> older kernels that's worth testing?

Not that I know of.  Probably not worth testing anything older than
that.

>> Perhaps we could conditionally stop trusting file ids sent by the server
>> as we currently do for hardlinks when we see these reparse points as
>> well.
>
> Maybe fail chdir with EXDEV unless mount is with noserverino?

That could be done, yes.

I'll try to follow your steps to get something set up on my Windows
server and see how both behave.  That will be useful for buildbot
testing as well.

Thanks for all the info.

Re: Nested NTFS volumes within Windows SMB share may result in inode collisions in linux client

[Paulo Alcantara]

Andrew Walker <awalker@ixsystems.com> writes:

> On my Windows server I mounted multiple NTFS volumes within the same
> share and played around until I was able to create directories with
> the same fileid number.

Could you try

      https://git.samba.org/sfrench/cifs-2.6.git for-next

and see if you can still reproduce duplicate fileids?

Thanks.