XFS, NFS and inode64 on 2.6.27

XFS, NFS and inode64 on 2.6.27

[Michael Monnerie]

[-- Attachment #1: Type: text/plain, Size: 2560 bytes --]

(After sending this to the XFS list, I was suggested by  
Christoph Hellwig to ask on this list. I'm not subscribed here, so 
please CC me)

Dear all,

after searching for a long time I guess I found that inode64 on XFS with 
kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. 
Sorry if this was already know, it wasn't for me and finding the problem 
was complicated.

This is my nfs4 root (fsid=0):

# ls -li
insgesamt 0
4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten
4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2
4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images
4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup
4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q
4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z

Then I mount those subdirs (mount --bind ....) to fill it with the dirs 
I want to share:

# ls -li
insgesamt 20
        256 drwxr-xr-x 14 root root  4096 19. Nov 03:01 daten
 6454034730 drwxrwx---  3 zmi  users   61 26. Mär 2009  download2
    4070671 drwxr-xr-x  2 root root    76 13. Nov 12:17 dvd-images
 6500330752 drwxr-xr-x  5 root root  4096  8. Nov 04:31 mailsrv-backup
 8591091465 drwxr-xr-x 17 root root  4096 12. Nov 02:01 q
 2147483904 drwxrwx--- 31 zmi  bh    4096  7. Nov 09:58 z

The shares "daten" and "dvd-images" can be mounted from other servers. I 
simply went to the original dirs of this mount-bind, and created several 
new dirs:
mkdir 1 2 3 4 5 6 7 8 9
and one of them had an inode < 2G, so I moved the contents there and 
renamed the dirs, remounted the --bind mounts and now have this:

# ls -li
insgesamt 20
      256 drwxr-xr-x 14 root root 4096 19. Nov 07:22 daten
184637244 drwxr-xr-x  3 root root   61 19. Nov 07:30 download2
  4070671 drwxr-xr-x  2 root root   76 13. Nov 12:17 dvd-images
960533297 drwxr-xr-x  5 root root 4096 19. Nov 07:27 mailsrv-backup
184190222 drwxr-xr-x 16 root root 4096 19. Nov 07:41 q
184637237 drwxr-xr-x 30 root root 4096 19. Nov 07:42 z

And everything mounts fine now.

(I still have one server that can mount, but not write, to such a share, 
but that may be another problem)

Just to let others know that can be a problem.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: XFS, NFS and inode64 on 2.6.27

[J. Bruce Fields]

On Mon, Nov 23, 2009 at 02:19:50PM +0100, Michael Monnerie wrote:
> after searching for a long time I guess I found that inode64 on XFS w=
ith=20
> kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS.=20
> Sorry if this was already know, it wasn't for me and finding the prob=
lem=20
> was complicated.
>=20
> This is my nfs4 root (fsid=3D0):
>=20
> # ls -li
> insgesamt 0
> 4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten
> 4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2
> 4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images
> 4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup
> 4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q
> 4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z
>=20
> Then I mount those subdirs (mount --bind ....) to fill it with the di=
rs=20
> I want to share:
>=20
> # ls -li
> insgesamt 20
>         256 drwxr-xr-x 14 root root  4096 19. Nov 03:01 daten
>  6454034730 drwxrwx---  3 zmi  users   61 26. M=C3=A4r 2009  download=
2
>     4070671 drwxr-xr-x  2 root root    76 13. Nov 12:17 dvd-images
>  6500330752 drwxr-xr-x  5 root root  4096  8. Nov 04:31 mailsrv-backu=
p
>  8591091465 drwxr-xr-x 17 root root  4096 12. Nov 02:01 q
>  2147483904 drwxrwx--- 31 zmi  bh    4096  7. Nov 09:58 z
>=20
> The shares "daten" and "dvd-images" can be mounted from other servers=
=2E

But not the other four directories?  What error do you get when you try=
?
What client are you using, and are you mounting with NFSv2, NFSv3, or
NFSv4?  Could we see a network trace of a failed mount?  (So, run

	tcpdump -s0 -wtmp.pcap

then attempt to mount one of the directories with a too-big inode
number, then kill tcpdump and send us tmp.pcap.)

--b.

Re: XFS, NFS and inode64 on 2.6.27

[Christoph Hellwig]

On Mon, Nov 23, 2009 at 10:51:09AM -0500, J. Bruce Fields wrote:
> But not the other four directories?  What error do you get when you try?
> What client are you using, and are you mounting with NFSv2, NFSv3, or
> NFSv4?  Could we see a network trace of a failed mount?  (So, run

>From information Michael posted on the xfs list he is not exporting the
root of filesystems, which means we use a file system handle format that
needs to encode the inode.  And all but the uuid16 format can only
encode 32 bit inode numbers.  I very strongly suspect that this is the
underlying issue here, but I'm currenly not uptodate on the filesystem
handle selection in nfsutils.

Re: XFS, NFS and inode64 on 2.6.27

[J. Bruce Fields]

On Wed, Nov 25, 2009 at 04:13:24PM -0500, Christoph Hellwig wrote:
> On Mon, Nov 23, 2009 at 10:51:09AM -0500, J. Bruce Fields wrote:
> > But not the other four directories?  What error do you get when you try?
> > What client are you using, and are you mounting with NFSv2, NFSv3, or
> > NFSv4?  Could we see a network trace of a failed mount?  (So, run
> 
> >From information Michael posted on the xfs list he is not exporting the
> root of filesystems, which means we use a file system handle format that
> needs to encode the inode.  And all but the uuid16 format can only
> encode 32 bit inode numbers.  I very strongly suspect that this is the
> underlying issue here, but I'm currenly not uptodate on the filesystem
> handle selection in nfsutils.

If using an nfs-utils recent enough to pass down a uuid (can check this
by looking at /proc/net/rpc/nfsd.export/filehandle after trying to
access those directories, and seeing whether there's a uuid= option
set), then I'd expect it to be using FSID_UUID16_INUM with v3 or v4 (v2
doesn't have large enough filehandles).

--b.

XFS, NFS and inode64 on 2.6.27

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 2415 bytes --]

Dear all,

after searching for a long time I guess I found that inode64 on XFS with 
kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. 
Sorry if this was already know, it wasn't for me and finding the problem 
was complicated.

This is my nfs4 root (fsid=0):

# ls -li
insgesamt 0
4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten
4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2
4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images
4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup
4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q
4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z

Then I mount those subdirs (mount --bind ....) to fill it with the dirs 
I want to share:

# ls -li
insgesamt 20
        256 drwxr-xr-x 14 root root  4096 19. Nov 03:01 daten
 6454034730 drwxrwx---  3 zmi  users   61 26. Mär 2009  download2
    4070671 drwxr-xr-x  2 root root    76 13. Nov 12:17 dvd-images
 6500330752 drwxr-xr-x  5 root root  4096  8. Nov 04:31 mailsrv-backup
 8591091465 drwxr-xr-x 17 root root  4096 12. Nov 02:01 q
 2147483904 drwxrwx--- 31 zmi  bh    4096  7. Nov 09:58 z

The shares "daten" and "dvd-images" can be mounted from other servers. I 
simply went to the original dirs of this mount-bind, and created several 
new dirs:
mkdir 1 2 3 4 5 6 7 8 9
and one of them had an inode < 2G, so I moved the contents there and 
renamed the dirs, remounted the --bind mounts and now have this:

# ls -li
insgesamt 20
      256 drwxr-xr-x 14 root root 4096 19. Nov 07:22 daten
184637244 drwxr-xr-x  3 root root   61 19. Nov 07:30 download2
  4070671 drwxr-xr-x  2 root root   76 13. Nov 12:17 dvd-images
960533297 drwxr-xr-x  5 root root 4096 19. Nov 07:27 mailsrv-backup
184190222 drwxr-xr-x 16 root root 4096 19. Nov 07:41 q
184637237 drwxr-xr-x 30 root root 4096 19. Nov 07:42 z

And everything mounts fine now.

(I still have one server that can mount, but not write, to such a share, 
but that may be another problem)

Just to let others know that can be a problem.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, NFS and inode64 on 2.6.27

[Nathaniel W. Turner]

Michael Monnerie wrote:
> after searching for a long time I guess I found that inode64 on XFS with 
> kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. 
> Sorry if this was already know, it wasn't for me and finding the problem 
> was complicated.
>   
Interesting.  I've never tried this, but I've read things that imply it 
should work.
Just out of curiosity, is your NFS client also running a 64-bit kernel?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, NFS and inode64 on 2.6.27

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 691 bytes --]

On Freitag, 20. November 2009 Nathaniel W. Turner wrote:
> Just out of curiosity, is your NFS client also running a 64-bit
>  kernel?

Yep. I've used 64bit since years, as AMD has that since a long time in 
all their CPUs (there have just been some Semprons 32bit only). And now 
since every PC has >3GB RAM it's nearly a must.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, NFS and inode64 on 2.6.27

[Christoph Hellwig]

On Fri, Nov 20, 2009 at 11:52:43AM +0100, Michael Monnerie wrote:
> The shares "daten" and "dvd-images" can be mounted from other servers. I 
> simply went to the original dirs of this mount-bind, and created several 
> new dirs:
> mkdir 1 2 3 4 5 6 7 8 9
> and one of them had an inode < 2G, so I moved the contents there and 
> renamed the dirs, remounted the --bind mounts and now have this:

So your NFS exports are not the roots of their respsective filesystems?
This means NFSD uses non-standard filesystem IDs in the filehandles
which have to encode the inode number of the export root.  Your best
option is to simplify switch to exporting a whole filesystem,
alternatively you can try making sure NFSD uses the 16byte wide UUID
style export.  Note thast either way will only work with a 64bit kernel
as the fs has no say in encoding the filesystem part of the handle
and ino_t is always 32bit on 32bit platforms.  This will also affect
any other filesystem with 64bit inode numbers.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, NFS and inode64 on 2.6.27

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1951 bytes --]

On Samstag, 21. November 2009 Christoph Hellwig wrote:
> So your NFS exports are not the roots of their respsective
>  filesystems? 

That's the way it works with NFSv4. You create an NFS root dir with a 
special entry in /etc/exports:
/nfsserver 10.0.0.0/8(fsid=0,rw,insecure,no_subtree_check,crossmnt)

and if you want other parts exported, create a dir there and mount it:
mkdir /nfsserver/data
mount --bind /mydata /nfsserver/data

This just takes the inodes from there and creates something like a view 
into this dir. From the client, you

mount -t nfs4 server:/data /serverdata
and you get the /nfsserver/data mounted there.

>  This means NFSD uses non-standard filesystem IDs in the
>  filehandles which have to encode the inode number of the export
>  root.  Your best option is to simplify switch to exporting a whole
>  filesystem, 

That's how it worked until NFSv3. I tried that also, didn't work either.
The whole thing worked before I reinstalled it (this server now runs 
virtualized within XENserver), and maybe that made some subdirs have 
inodes > 32bit.

>  alternatively you can try making sure NFSD uses the
>  16byte wide UUID style export.  

How'd I do that?

>  Note thast either way will only work
>  with a 64bit kernel as the fs has no say in encoding the filesystem
>  part of the handle and ino_t is always 32bit on 32bit platforms. 
>  This will also affect any other filesystem with 64bit inode numbers.

Not my problem, everything 64bit here. I use the kernel-nfsd, and that's 
why I was wondering to have this problem.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, NFS and inode64 on 2.6.27

[Christoph Hellwig]

On Sat, Nov 21, 2009 at 10:17:57PM +0100, Michael Monnerie wrote:
> >  filehandles which have to encode the inode number of the export
> >  root.  Your best option is to simplify switch to exporting a whole
> >  filesystem, 
> 
> That's how it worked until NFSv3. I tried that also, didn't work either.
> The whole thing worked before I reinstalled it (this server now runs 
> virtualized within XENserver), and maybe that made some subdirs have 
> inodes > 32bit.
> 
> >  alternatively you can try making sure NFSD uses the
> >  16byte wide UUID style export.  
> 
> How'd I do that?

I'm not sure, haven't deal with nfs utils a lot recently.  I'd suggest
you brings this up on linux-nfs@vger.kernel.org.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs