XFS: inode with st_mode == 0

XFS: inode with st_mode == 0

[Jan Kasprzak]

	Hi all,

I have seen the strange problem on our NFS server: yesterday I have
found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home
directory (ls -l said "?--------- 1 root root 0 <date> <filename>").
The <filename> was correct name of a temporary file used by one of my
cron jobs (and the cron job was failing because it could not rewrite the file).
It was not possible to write to this file, so I have renamed it
as "badfile" for further investigation (using mv(1) on the NFS server itself).

Today I did "ls -l badfile", and the file had correct owner/group
and st_mode as expected from file created by my cron job: 

# stat badfile
  File: `badfile'
  Size: 0               Blocks: 0          IO Block: 4096   Regular File
Device: fd00h/64768d    Inode: 356913      Links: 1
Access: (0644/-rw-r--r--)  Uid: (xxxx/     kas)   Gid: (yyyy/ student)
Access: 2004-12-07 21:17:01.000000000 +0100
Modify: 2004-12-07 21:17:01.000000000 +0100

We have seen empty files with uid==gid==0 and st_mode==0 few times
before, at places where newly-created regular files were expected,
but we have simply deleted them. This time I have renamed the file
and looked at it after a day or so.

These files were created over NFS (our home directories are served by
NFS server) from various NFS clients (this one was Solaris 8, at least
one of the previous instances was created by an IRIX NFS client). The
filesystem itself is XFS on a LVM2 volume. The server is HP DL-585 quad
opteron, running RHEL3 plus our 2.6.10-rc2 kernel.

Maybe some data is flushed in an incorrect order?

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Thu, Dec 09, 2004 at 01:59:18PM +0100, Jan Kasprzak wrote:
> 	Hi all,
> 
> I have seen the strange problem on our NFS server: yesterday I have
> found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home
> directory (ls -l said "?--------- 1 root root 0 <date> <filename>").
> The <filename> was correct name of a temporary file used by one of my
> cron jobs (and the cron job was failing because it could not rewrite the file).
> It was not possible to write to this file, so I have renamed it
> as "badfile" for further investigation (using mv(1) on the NFS server itself).

Known problem

http://lkml.org/lkml/2004/11/23/283

Seems there is no solution yet

http://lkml.org/lkml/2004/11/30/145

...
> 
> Maybe some data is flushed in an incorrect order?

Maybe  :)

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Jan Kasprzak]

Jakob Oestergaard wrote:
: > I have seen the strange problem on our NFS server: yesterday I have
: > found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home
: > directory (ls -l said "?--------- 1 root root 0 <date> <filename>").
: > The <filename> was correct name of a temporary file used by one of my
: > cron jobs (and the cron job was failing because it could not rewrite the file).
: > It was not possible to write to this file, so I have renamed it
: > as "badfile" for further investigation (using mv(1) on the NFS server itself).
: 
: Known problem
: 
: http://lkml.org/lkml/2004/11/23/283
: 
: Seems there is no solution yet
: 
: http://lkml.org/lkml/2004/11/30/145
: 
	Hmm, thanks for the info.

	In the meantime, I was able to reproduce this by running the
following script for an hour or so on Solaris 8 NFS client:

#!/bin/bash
FILE="testfile"
set -e
while :
do
        rm -f "$FILE"
        sleep 1
        echo trubka > "$FILE"
        sleep 1
        cat "$FILE">/dev/null
done

After some time I've got this:

emptyfile.sh: testfile: Invalid argument
$ ls -l testfile
?---------   1 root     root           0 Dec  9 14:51 testfile

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <

Re: XFS: inode with st_mode == 0

[Christoph Hellwig]

[sorry, last post got out to far too few people]

On Thu, Dec 09, 2004 at 02:53:23PM +0100, Jakob Oestergaard wrote:
> On Thu, Dec 09, 2004 at 01:59:18PM +0100, Jan Kasprzak wrote:
> > 	Hi all,
> > 
> > I have seen the strange problem on our NFS server: yesterday I have
> > found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home
> > directory (ls -l said "?--------- 1 root root 0 <date> <filename>").
> > The <filename> was correct name of a temporary file used by one of my
> > cron jobs (and the cron job was failing because it could not rewrite the file).
> > It was not possible to write to this file, so I have renamed it
> > as "badfile" for further investigation (using mv(1) on the NFS server itself).
> 
> Known problem
> 
> http://lkml.org/lkml/2004/11/23/283
> 
> Seems there is no solution yet
> 
> http://lkml.org/lkml/2004/11/30/145

If it's really st_mode I suspect it's a different problem.  Can you retry
with current oss.sgi.com CVS (or the patch below).  Note that this patch
breaks xfsdump unfortunately, we're looking into a fix.

> > Maybe some data is flushed in an incorrect order?
> 
> Maybe  :)

No, the problem I've fixed was related to XFS getting the inode version
number wrong - or at least different than NFSD expects.


Index: fs/xfs/xfs_ialloc.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_ialloc.c,v
retrieving revision 1.175
diff -u -p -r1.175 xfs_ialloc.c
--- fs/xfs/xfs_ialloc.c	6 Oct 2003 18:11:55 -0000	1.175
+++ fs/xfs/xfs_ialloc.c	30 Nov 2004 10:18:16 -0000
@@ -270,6 +270,11 @@ xfs_ialloc_ag_alloc(
 	INT_SET(dic.di_magic, ARCH_CONVERT, XFS_DINODE_MAGIC);
 	INT_SET(dic.di_version, ARCH_CONVERT, version);
 
+	/*
+	 * Start at generation 1 because the NFS code uses 0 as wildcard.
+	 */
+	INT_SET(dic.di_gen, ARCH_CONVERT, 1);
+
 	for (j = 0; j < nbufs; j++) {
 		/*
 		 * Get the block.
Index: fs/xfs/xfs_inode.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_inode.c,v
retrieving revision 1.406
diff -u -p -r1.406 xfs_inode.c
--- fs/xfs/xfs_inode.c	27 Oct 2004 12:06:24 -0000	1.406
+++ fs/xfs/xfs_inode.c	30 Nov 2004 10:18:18 -0000
@@ -1028,6 +1028,16 @@ xfs_iread(
 		ip->i_d.di_projid = 0;
 	}
 
+	/*
+	 * IRIX and older Linux Kernel initialized di_gen to zero when
+	 * creating new inodes, but the NFSD uses i_generation = 0 as
+	 * a wildcard.  We bump di_gen here to avoid that problem for new
+	 * exports, and filehandles created by an older kernel using
+	 * the same filesystem are still valid as the wildcard matches it.
+	 */
+	if (unlikely(ip->i_d.di_gen == 0))
+		ip->i_d.di_gen++;
+
 	ip->i_delayed_blks = 0;
 
 	/*
@@ -2370,11 +2380,17 @@ xfs_ifree(
 		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
 	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
 	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
+
 	/*
 	 * Bump the generation count so no one will be confused
-	 * by reincarnations of this inode.
+	 * by reincarnations of this inode.  Note that we have to
+	 * skip 0 as that would confuse the NFS server, and we
+	 * have to do it here because the XFS inode might be
+	 * around for a while after deletion (unlike the normal
+	 * Linux inode semantics).
 	 */
-	ip->i_d.di_gen++;
+	if (++ip->i_d.di_gen == 0)
+		ip->i_d.di_gen++;
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 
 	if (delete) {

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Thu, Dec 09, 2004 at 09:54:14PM +0000, Christoph Hellwig wrote:
...
> 
> If it's really st_mode I suspect it's a different problem.  Can you retry
> with current oss.sgi.com CVS (or the patch below).  Note that this patch
> breaks xfsdump unfortunately, we're looking into a fix.

I'm running plain 2.6.9 plus the patch you sent now.

So far, the server hasn't hosed itself (and it's been running for almost
half an hour - woohoo)

Now I can check out CVS trees without getting files with weird ownership
and undeletable directories  (which as *extremely* common before - in
fact, earlier today before I applied the patch, I was unable to complete
a cvs checkout of a moderately large software project).

I guess this is a thumbs up for your patch so far  :)


> > > Maybe some data is flushed in an incorrect order?
> > 
> > Maybe  :)
> 
> No, the problem I've fixed was related to XFS getting the inode version
> number wrong - or at least different than NFSD expects.

I have seen problems that your patch probably won't fix (like
undeletable directories randomly occuring on ext3 filesystems as well -
seems that once the dcache gets confused, it *really* gets confused).

Let's see how the box fares with your patch - I'll report back on new
bugs as I find them.

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Jan Kasprzak]

Christoph Hellwig wrote:
: > On Thu, Dec 09, 2004 at 01:59:18PM +0100, Jan Kasprzak wrote:
: > > I have seen the strange problem on our NFS server: yesterday I have
: > > found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home
: > > directory (ls -l said "?--------- 1 root root 0 <date> <filename>").
[...]
: If it's really st_mode I suspect it's a different problem.  Can you retry
: with current oss.sgi.com CVS (or the patch below).  Note that this patch
: breaks xfsdump unfortunately, we're looking into a fix.
: 
: > > Maybe some data is flushed in an incorrect order?
: > 
: > Maybe  :)
: 
: No, the problem I've fixed was related to XFS getting the inode version
: number wrong - or at least different than NFSD expects.
: 
	We have applied these two patches to 2.6.10-rc2, but this
does not help. A few minutes ago I've got the "?----------" file
again from my test script. This time it took >4 hours (it was
about an hour or so without this patch).

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Tue, Dec 21, 2004 at 07:43:04PM +0100, Jan Kasprzak wrote:
...
> : 
> : No, the problem I've fixed was related to XFS getting the inode version
> : number wrong - or at least different than NFSD expects.
> : 
> 	We have applied these two patches to 2.6.10-rc2, but this
> does not help. A few minutes ago I've got the "?----------" file
> again from my test script. This time it took >4 hours (it was
> about an hour or so without this patch).

I run the patch on 2.6.9 - it solved the problem in the common case.
Before the patch, I was unable to complete a "cvs checkout" of a
moderately large tree - would end up with undeletable directories and
lots of other weird things...  After the patch, I can run cvs checkout.

However, we still see the problem - so the patch does not solve this
completely, as you have observed as well.

Our most common situation is that a new file gets created as a symlink
pointing to itself, instead of as a regular file.

It also happens regularly, that a command (be it cvs, etags, ld or
something else) reports EINVAL when attempting to create/write a file.

So, status on my side is;  things still suck, but they suck less than on
vanilla 2.6.9

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Christoph Hellwig]

On Wed, Dec 22, 2004 at 09:41:58AM +0100, Jakob Oestergaard wrote:
> On Tue, Dec 21, 2004 at 07:43:04PM +0100, Jan Kasprzak wrote:
> ...
> > : 
> > : No, the problem I've fixed was related to XFS getting the inode version
> > : number wrong - or at least different than NFSD expects.
> > : 
> > 	We have applied these two patches to 2.6.10-rc2, but this
> > does not help. A few minutes ago I've got the "?----------" file
> > again from my test script. This time it took >4 hours (it was
> > about an hour or so without this patch).
> 
> I run the patch on 2.6.9 - it solved the problem in the common case.
> Before the patch, I was unable to complete a "cvs checkout" of a
> moderately large tree - would end up with undeletable directories and
> lots of other weird things...  After the patch, I can run cvs checkout.
> 
> However, we still see the problem - so the patch does not solve this
> completely, as you have observed as well.
> 
> Our most common situation is that a new file gets created as a symlink
> pointing to itself, instead of as a regular file.
> 
> It also happens regularly, that a command (be it cvs, etags, ld or
> something else) reports EINVAL when attempting to create/write a file.
> 
> So, status on my side is;  things still suck, but they suck less than on
> vanilla 2.6.9

I have a better patch than the one I gave you (attached below).  If you
send me a mail with steps to reproduce your remaining problems I'll put
this very high on my TODO list after christmas.  Btw, any chance you could
try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
there have been various other fixes in the last months.


Index: fs/xfs/xfs_vfsops.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_vfsops.c,v
retrieving revision 1.459
diff -u -p -r1.459 xfs_vfsops.c
--- fs/xfs/xfs_vfsops.c	15 Dec 2004 04:56:58 -0000	1.459
+++ fs/xfs/xfs_vfsops.c	16 Dec 2004 20:47:22 -0000
@@ -1581,7 +1581,7 @@ xfs_syncsub(
 }
 
 /*
- * xfs_vget - called by DMAPI to get vnode from file handle
+ * xfs_vget - called by DMAPI and NFSD to get vnode from file handle
  */
 STATIC int
 xfs_vget(
@@ -1623,7 +1623,7 @@ xfs_vget(
 		return XFS_ERROR(EIO);
 	}
 
-	if (ip->i_d.di_mode == 0 || (igen && (ip->i_d.di_gen != igen))) {
+	if (ip->i_d.di_mode == 0 || ip->i_d.di_gen != igen) {
 		xfs_iput_new(ip, XFS_ILOCK_SHARED);
 		*vpp = NULL;
 		return XFS_ERROR(ENOENT);
Index: fs/xfs/linux-2.6/xfs_super.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_super.c,v
retrieving revision 1.321
diff -u -p -r1.321 xfs_super.c
--- fs/xfs/linux-2.6/xfs_super.c	9 Dec 2004 02:41:20 -0000	1.321
+++ fs/xfs/linux-2.6/xfs_super.c	16 Dec 2004 20:47:23 -0000
@@ -731,6 +731,39 @@ linvfs_get_dentry(
 	return result;
 }
 
+STATIC struct dentry *
+linvfs_decode_fh(
+	struct super_block	*sb,
+	__u32			*fh,
+	int			fh_len,
+	int			fileid_type,
+	int (*acceptable)(
+		void		*context,
+		struct dentry	*de),
+	void			*context)
+{
+	__u32 parent[2];
+	parent[0] = parent[1] = 0;
+	
+	if (fh_len < 2 || fileid_type > 2)
+		return NULL;
+	
+	if (fileid_type == 2 && fh_len > 2) {
+		if (fh_len == 3) {
+			printk(KERN_WARNING
+			       "XFS: detected filehandle without "
+			       "parent inode generation information.");
+			return ERR_PTR(-ESTALE);
+		}
+			
+		parent[0] = fh[2];
+		parent[1] = fh[3];
+	}
+	
+	return find_exported_dentry(sb, fh, parent, acceptable, context);
+
+}
+
 STATIC int
 linvfs_show_options(
 	struct seq_file		*m,
@@ -893,6 +926,7 @@ linvfs_get_sb(
 
 
 STATIC struct export_operations linvfs_export_ops = {
+	.decode_fh		= linvfs_decode_fh,
 	.get_parent		= linvfs_get_parent,
 	.get_dentry		= linvfs_get_dentry,
 };
Index: include/linux/fs.h
===================================================================
RCS file: /cvs/linux-2.6-xfs/include/linux/fs.h,v
retrieving revision 1.11
diff -u -p -r1.11 fs.h
--- include/linux/fs.h	1 Oct 2004 15:10:15 -0000	1.11
+++ include/linux/fs.h	16 Dec 2004 20:47:25 -0000
@@ -1115,6 +1115,10 @@ struct export_operations {
 
 };
 
+extern struct dentry *
+find_exported_dentry(struct super_block *sb, void *obj, void *parent,
+		     int (*acceptable)(void *context, struct dentry *de),
+		     void *context);
 
 struct file_system_type {
 	const char *name;

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Wed, Dec 22, 2004 at 06:23:44PM +0000, Christoph Hellwig wrote:
...
> I have a better patch than the one I gave you (attached below).  If you
> send me a mail with steps to reproduce your remaining problems I'll put
> this very high on my TODO list after christmas.  Btw, any chance you could
> try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
> there have been various other fixes in the last months.

Thank you Christoph,

I will see what I can do - my schedule is a little tight these days, but
I'll try it out at the first given opportunity (which will definitely be
after christmas but hopefully before new year).

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Wed, Dec 22, 2004 at 06:23:44PM +0000, Christoph Hellwig wrote:
...
> I have a better patch than the one I gave you (attached below).  If you
> send me a mail with steps to reproduce your remaining problems I'll put
> this very high on my TODO list after christmas.  Btw, any chance you could
> try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
> there have been various other fixes in the last months.

I have been on XFS CVS + the patch you sent for five days now.

Summary:

 XFS and related dcache problems (seen in ext3 too) with files that get
created as symlinks to themselves or undeletable directories, or just
get plain wrong permissions or ownership:   These problems seem to have
gone away completely.   Very very nice!

 I still get NFS stale handle problems (the weird ones that can be
worked around at times by running a ls -l on the server-side).

 I talked Anders (as@cohaesio.com) into testing the same kernel on
another system he's running, where his main prolem were the stale
handles. He too now only sees the weird stale handle problems.

So, it looks like the current 2.6 status on file serving is:
 *) Can be done with XFS+NFS+SMP iff running a SGI kernel
 *) Will see weird NFS stale handle problems no matter which kernel
    is used

Which is a lot better than the previous "2.6 will eat your files and
make whatever remains owned by guest" situation   :)

Good work!

(Any suggestions on the knfsd issue with stale handles?)

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Christoph Hellwig]

On Tue, Jan 04, 2005 at 09:48:10AM +0100, Jakob Oestergaard wrote:
> (Any suggestions on the knfsd issue with stale handles?)

I'd suggest asking nfs@lists.sourceforget.net for help, tell them your
exact setup - OS rev on client/server, mount options /etc/exports options,
etc..

I'm a bit lost in NFS deep magic land still :)

Re: XFS: inode with st_mode == 0

[David Greaves]

Christoph Hellwig wrote:

>>>	We have applied these two patches to 2.6.10-rc2, but this
>>>does not help. A few minutes ago I've got the "?----------" file
>>>again from my test script. This time it took >4 hours (it was
>>>about an hour or so without this patch).
>>>      
>>>
I'm seeing this problem occasionally too.

I'm running 2.6.10

The patch you provided below can be made to apply but I get:
LD .tmp_vmlinux1
fs/built-in.o(.text+0xa0ed3): In function `linvfs_decode_fh':
: undefined reference to `find_exported_dentry'

I assume that since you say:

> Btw, any chance you could
>try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
>there have been various other fixes in the last months.
>  
>
That not all the changes in XFS CVS have made it to 2.6.10?

Is there a 2.6.10 patch that I could apply? Or do you have any other 
suggestions.

David

>
>Index: fs/xfs/xfs_vfsops.c
>===================================================================
>RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_vfsops.c,v
>retrieving revision 1.459
>diff -u -p -r1.459 xfs_vfsops.c
>--- fs/xfs/xfs_vfsops.c	15 Dec 2004 04:56:58 -0000	1.459
>+++ fs/xfs/xfs_vfsops.c	16 Dec 2004 20:47:22 -0000
>@@ -1581,7 +1581,7 @@ xfs_syncsub(
> }
> 
> /*
>- * xfs_vget - called by DMAPI to get vnode from file handle
>+ * xfs_vget - called by DMAPI and NFSD to get vnode from file handle
>  */
> STATIC int
> xfs_vget(
>@@ -1623,7 +1623,7 @@ xfs_vget(
> 		return XFS_ERROR(EIO);
> 	}
> 
>-	if (ip->i_d.di_mode == 0 || (igen && (ip->i_d.di_gen != igen))) {
>+	if (ip->i_d.di_mode == 0 || ip->i_d.di_gen != igen) {
> 		xfs_iput_new(ip, XFS_ILOCK_SHARED);
> 		*vpp = NULL;
> 		return XFS_ERROR(ENOENT);
>Index: fs/xfs/linux-2.6/xfs_super.c
>===================================================================
>RCS file: /cvs/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_super.c,v
>retrieving revision 1.321
>diff -u -p -r1.321 xfs_super.c
>--- fs/xfs/linux-2.6/xfs_super.c	9 Dec 2004 02:41:20 -0000	1.321
>+++ fs/xfs/linux-2.6/xfs_super.c	16 Dec 2004 20:47:23 -0000
>@@ -731,6 +731,39 @@ linvfs_get_dentry(
> 	return result;
> }
> 
>+STATIC struct dentry *
>+linvfs_decode_fh(
>+	struct super_block	*sb,
>+	__u32			*fh,
>+	int			fh_len,
>+	int			fileid_type,
>+	int (*acceptable)(
>+		void		*context,
>+		struct dentry	*de),
>+	void			*context)
>+{
>+	__u32 parent[2];
>+	parent[0] = parent[1] = 0;
>+	
>+	if (fh_len < 2 || fileid_type > 2)
>+		return NULL;
>+	
>+	if (fileid_type == 2 && fh_len > 2) {
>+		if (fh_len == 3) {
>+			printk(KERN_WARNING
>+			       "XFS: detected filehandle without "
>+			       "parent inode generation information.");
>+			return ERR_PTR(-ESTALE);
>+		}
>+			
>+		parent[0] = fh[2];
>+		parent[1] = fh[3];
>+	}
>+	
>+	return find_exported_dentry(sb, fh, parent, acceptable, context);
>+
>+}
>+
> STATIC int
> linvfs_show_options(
> 	struct seq_file		*m,
>@@ -893,6 +926,7 @@ linvfs_get_sb(
> 
> 
> STATIC struct export_operations linvfs_export_ops = {
>+	.decode_fh		= linvfs_decode_fh,
> 	.get_parent		= linvfs_get_parent,
> 	.get_dentry		= linvfs_get_dentry,
> };
>Index: include/linux/fs.h
>===================================================================
>RCS file: /cvs/linux-2.6-xfs/include/linux/fs.h,v
>retrieving revision 1.11
>diff -u -p -r1.11 fs.h
>--- include/linux/fs.h	1 Oct 2004 15:10:15 -0000	1.11
>+++ include/linux/fs.h	16 Dec 2004 20:47:25 -0000
>@@ -1115,6 +1115,10 @@ struct export_operations {
> 
> };
> 
>+extern struct dentry *
>+find_exported_dentry(struct super_block *sb, void *obj, void *parent,
>+		     int (*acceptable)(void *context, struct dentry *de),
>+		     void *context);
> 
> struct file_system_type {
> 	const char *name;
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Fri, Jan 14, 2005 at 06:14:55PM +0000, David Greaves wrote:
> 
...
> >try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
> >there have been various other fixes in the last months.
> > 
> >
> That not all the changes in XFS CVS have made it to 2.6.10?
> 
> Is there a 2.6.10 patch that I could apply? Or do you have any other 
> suggestions.

AFAIK the best you can do is to get the most recent XFS kernel from
SGI's CVS (this one is based on 2.6.10).

If you run that kernel, then most of the former problems will be gone;
*) I only have one undeletable directory on my system - so it seems that
this error is no longer common   ;)
*) 2.6.10 apparently fixes the knfsd stale handle problem
*) I no longer see the weird directory/file/??? mode problems or random
ownership assignment

So apart from the general well known instability problems that will
occur when you actually start *using* the system, there should be no
major problems running XFS+SMP+NFS on SGI's 2.6.10 based kernel   ;)

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Nathan Scott]

On Fri, Jan 14, 2005 at 07:23:09PM +0100, Jakob Oestergaard wrote:
> > Is there a 2.6.10 patch that I could apply? Or do you have any other 
> > suggestions.
> 
> AFAIK the best you can do is to get the most recent XFS kernel from
> SGI's CVS (this one is based on 2.6.10).

The -mm tree also has these fixes; we'll get them merged into
mainline soon.

> If you run that kernel, then most of the former problems will be gone;
> *) I only have one undeletable directory on my system - so it seems that
> this error is no longer common   ;)

You may need to run xfs_repair to clean that up..?  Or does
the problem persist after a repair?

cheers.

-- 
Nathan

Re: XFS: inode with st_mode == 0

[Christoph Hellwig]

On Fri, Jan 14, 2005 at 07:23:09PM +0100, Jakob Oestergaard wrote:
> So apart from the general well known instability problems that will
> occur when you actually start *using* the system, there should be no

What known instabilities?

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Sat, Jan 15, 2005 at 01:09:08PM +1100, Nathan Scott wrote:
...
> > AFAIK the best you can do is to get the most recent XFS kernel from
> > SGI's CVS (this one is based on 2.6.10).
> 
> The -mm tree also has these fixes; we'll get them merged into
> mainline soon.

Okeydokey - good

> 
> > If you run that kernel, then most of the former problems will be gone;
> > *) I only have one undeletable directory on my system - so it seems that
> > this error is no longer common   ;)
> 
> You may need to run xfs_repair to clean that up..?  Or does
> the problem persist after a repair?

I'm running Debian Woody - the xfs_check/xfs_repair there didn't seem to
find anything last I tried. I have not re-checked for this last problem
though.

I figured I might need to run the CVS version of xfs tools, and, well,
me being busy and all, I thought I'd just leave the 'delete_me'
directory hanging until some time I got more time on my hands  ;)

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Jakob Oestergaard]

On Sun, Jan 16, 2005 at 01:51:12PM +0000, Christoph Hellwig wrote:
> On Fri, Jan 14, 2005 at 07:23:09PM +0100, Jakob Oestergaard wrote:
> > So apart from the general well known instability problems that will
> > occur when you actually start *using* the system, there should be no
> 
> What known instabilities?

Where should I begin?  ;)

Most of the following have already been posted to LKML - primarily by
Anders (as@cohaesio.com) - it seems that noone cares, but I'll repost a
summary that Anders sent me below:

-------
Scenario 1: Mailservers:
  2.6.10 (~24-40 hours uptime):
  Running ext3 on mailqueue:

<SNIP>
Unable to handle kernel NULL pointer dereference at virtual address 00000004
printing eip:
c018a095
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: nfs e1000 iptable_nat ipt_connlimit rtc
CPU:    2
EIP:    0060:[<c018a095>]    Not tainted
EFLAGS: 00010286   (2.6.8.1)
EIP is at journal_commit_transaction+0x535/0x10e5
eax: cac1e26c   ebx: 00000000   ecx: f7cec400   edx: f7cec400
esi: f65f3000   edi: cac1e26c   ebp: f65f3000   esp: f65f3dc0
ds: 007b   es: 007b   ss: 0068
Process kjournald (pid: 174, threadinfo=f65f3000 task=c2308b70)
Stack: f65f3e64 00000000 00000000 00000000 00000000 00000000 f7cec400 cda565fc
       0000149a 00000004 f65f3e48 c01132d8 00000002 c202ad20 00000001 f65f3e5c
       c202ad20 c202ad20 00000002 00000001 0000001e 01c1af60 f65f3e68 c0407dc0
Call Trace:
 [<c01132d8>] scheduler_tick+0x468/0x470
 [<c01127b5>] find_busiest_group+0x105/0x310
 [<c011db8e>] del_timer_sync+0x7e/0xa0
 [<c018cd4d>] kjournald+0xbd/0x230
 [<c0114b10>] autoremove_wake_function+0x0/0x40
 [<c0114b10>] autoremove_wake_function+0x0/0x40
 [<c0103f16>] ret_from_fork+0x6/0x14
 [<c018cc70>] commit_timeout+0x0/0x10
 [<c018cc90>] kjournald+0x0/0x230
 [<c01024bd>] kernel_thread_helper+0x5/0x18
Code: f0 ff 43 04 8b 03 83 e0 04 74 4c 8b 8c 24 b8 01 00 00 c6 81
 <2>SoftDog: Initiating system reboot
</SNIP>

-------
Scenario 2: Mailservers:
  Running XFS on mailqueue:

<SNIP>
Filesystem "sdb1": xfs_trans_delete_ail: attempting to delete a log item that 
is not in the AIL
xfs_force_shutdown(sdb1,0x8) called from line 382 of file 
fs/xfs/xfs_trans_ail.c.  Return address = 0xc0216a56
@Linux version 2.6.9 (root@mail1.domain.tld) (gcc version 2.96 20000731 (Red 
Hat Linux 7.3 2.96-113)) #1 SMP Tue Oct 19 16:04:55 CEST 2004
</SNIP>


=======
Resolution to the mailserver problem:
 2.4.28 is perfectly stable on these machines.

-------
Scenario 3: Webservers:

  2.6.10, 2.6.10-ac8 (~3-12 hours uptime):

    <SNIP>
    Unable to handle kernel paging request
    <2>SoftDog: Initiating system reboot.
    <SNIP>
    (No more...) :(

=======
Resolution to the webserver problem:
 2.4.28/2.4.29-rc2 are stable here

-------
Scenario 4: Storageservers: 
  2.6.8.1:
    Oopses after ~5-10 hours whith SMP on. - Cannot find the actual Oopses 
anymore and 2.6.8+ havent been tested as we cannot afford anymore downtime on 
these servers.


=======
Resolution to the storage server problem:
 2.6.8.1 UP is stable (but oopses regularly after memory allocation
 failures)



Hardware on all servers: IBM x335 and x345.

Mentioned errors seen on a total of 17 servers.

-- 

 / jakob

Re: XFS: inode with st_mode == 0

[Jan-Frode Myklebust]

On Mon, Jan 17, 2005 at 11:07:46AM +0100, Jakob Oestergaard wrote:
> 
> Where should I begin?  ;)

Guess we've been struggeling with much of the same problems..

> -------
> Scenario 2: Mailservers:
>   Running XFS on mailqueue:

The 2.6.10-1.737_FC3 + 's/posix_lock_file/posix_lock_file_wait/' on
fs/nfs/file.c seems stable on our mailserver running XFS on
mail queue and spool (mbox). 4 days of uptime! 

> 
> =======
> Resolution to the storage server problem:
>  2.6.8.1 UP is stable (but oopses regularly after memory allocation
>  failures)

My XFS-fileserver ran 2.6.9-rc3 stable since october 25. Got lots of
"possible deadlock in kmem_alloc (mode:0xd0)" this weekend, so I
upgraded to plain 2.6.10. Seems OK so far. 

> 
> Hardware on all servers: IBM x335 and x345.

Mail servers: Dell 2650, IBM ServeRAID 6M, EXP400.
File servers: IBM x330, qla2300, infortrend eonstor.

All running Whitebox/centos RHEL clone.


  -jf

Re: XFS: inode with st_mode == 0

[Anders Saaby]

Hi,

On Monday 17 January 2005 12:55, Jan-Frode Myklebust wrote:
>
> Guess we've been struggeling with much of the same problems..

Seems like it. :)

> > -------
> > Scenario 2: Mailservers:
> >   Running XFS on mailqueue:
>
> The 2.6.10-1.737_FC3 + 's/posix_lock_file/posix_lock_file_wait/' on
> fs/nfs/file.c seems stable on our mailserver running XFS on
> mail queue and spool (mbox). 4 days of uptime!

Yes - We had those errors to:

"Kernel panic - not syncing: Attempting to free lock with active block list"

- on 2.6.10 on the webservers, which was fixed with that particular patch. But 
this is a different error as our mailservers dont't act as NFS clients. All 
use local XFS.

Sad thing is that the mailservers crashes every 10-20 hours on 2.6.x, but I'm 
not able to reproduce it in a test environment, and at time of original post 
to LKML noone was able to do anything about it without a reproduceable 
testcase. :(

> > =======
> > Resolution to the storage server problem:
> >  2.6.8.1 UP is stable (but oopses regularly after memory allocation
> >  failures)
>
> My XFS-fileserver ran 2.6.9-rc3 stable since october 25. Got lots of
> "possible deadlock in kmem_alloc (mode:0xd0)" this weekend, so I
> upgraded to plain 2.6.10. Seems OK so far.
>

OK, as far as i remember, we had the same messages in the kernel log when 
running with SMP.

-- 
Med venlig hilsen - Best regards - Meilleures salutations

Anders Saaby
Systems Engineer
------------------------------------------------
Cohaesio A/S - Maglebjergvej 5D - DK-2800 Lyngby
Phone: +45 45 880 888 - Fax: +45 45 880 777
Mail: as@cohaesio.com - http://www.cohaesio.com
------------------------------------------------

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Alan Cox]

On Llu, 2005-01-17 at 21:31, Jeffrey Hundstad wrote:
> I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few 
> hours on a machine under real load.   Perhaps us folks with the problem 
> need to talk to the powers who be to come up with a strategy to make a 
> report they can use.  My guess is we're not sending something that can 
> be used.

I need a way to reproduce it. Preferably on a hardware configuration
that is running 2.6.10-ac10 or later because of the bio and acpi fixes.
I'm not interested in any report including binary drivers and to be
honest the least complex configuration the better. I also care that the
hardware passes memtest86+ !

I also don't care about XFS although Christoph may well do.

Alan

journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Jeffrey Hundstad]

For more of this look up subjects:
  Bad things happening to journaled filesystem machines
  Oops in kjournald
and from author:
  Anders Saaby

I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few 
hours on a machine under real load.   Perhaps us folks with the problem 
need to talk to the powers who be to come up with a strategy to make a 
report they can use.  My guess is we're not sending something that can 
be used.

-- 
jeffrey hundstad


Jakob Oestergaard wrote:

>On Sun, Jan 16, 2005 at 01:51:12PM +0000, Christoph Hellwig wrote:
>  
>
>>On Fri, Jan 14, 2005 at 07:23:09PM +0100, Jakob Oestergaard wrote:
>>    
>>
>>>So apart from the general well known instability problems that will
>>>occur when you actually start *using* the system, there should be no
>>>      
>>>
>>What known instabilities?
>>    
>>
>
>Where should I begin?  ;)
>
>Most of the following have already been posted to LKML - primarily by
>Anders (as@cohaesio.com) - it seems that noone cares, but I'll repost a
>summary that Anders sent me below:
>
>-------
>Scenario 1: Mailservers:
>  2.6.10 (~24-40 hours uptime):
>  Running ext3 on mailqueue:
>
><SNIP>
>Unable to handle kernel NULL pointer dereference at virtual address 00000004
>printing eip:
>c018a095
>*pde = 00000000
>Oops: 0002 [#1]
>SMP
>Modules linked in: nfs e1000 iptable_nat ipt_connlimit rtc
>CPU:    2
>EIP:    0060:[<c018a095>]    Not tainted
>EFLAGS: 00010286   (2.6.8.1)
>EIP is at journal_commit_transaction+0x535/0x10e5
>eax: cac1e26c   ebx: 00000000   ecx: f7cec400   edx: f7cec400
>esi: f65f3000   edi: cac1e26c   ebp: f65f3000   esp: f65f3dc0
>ds: 007b   es: 007b   ss: 0068
>Process kjournald (pid: 174, threadinfo=f65f3000 task=c2308b70)
>Stack: f65f3e64 00000000 00000000 00000000 00000000 00000000 f7cec400 cda565fc
>       0000149a 00000004 f65f3e48 c01132d8 00000002 c202ad20 00000001 f65f3e5c
>       c202ad20 c202ad20 00000002 00000001 0000001e 01c1af60 f65f3e68 c0407dc0
>Call Trace:
> [<c01132d8>] scheduler_tick+0x468/0x470
> [<c01127b5>] find_busiest_group+0x105/0x310
> [<c011db8e>] del_timer_sync+0x7e/0xa0
> [<c018cd4d>] kjournald+0xbd/0x230
> [<c0114b10>] autoremove_wake_function+0x0/0x40
> [<c0114b10>] autoremove_wake_function+0x0/0x40
> [<c0103f16>] ret_from_fork+0x6/0x14
> [<c018cc70>] commit_timeout+0x0/0x10
> [<c018cc90>] kjournald+0x0/0x230
> [<c01024bd>] kernel_thread_helper+0x5/0x18
>Code: f0 ff 43 04 8b 03 83 e0 04 74 4c 8b 8c 24 b8 01 00 00 c6 81
> <2>SoftDog: Initiating system reboot
></SNIP>
>
>-------
>Scenario 2: Mailservers:
>  Running XFS on mailqueue:
>
><SNIP>
>Filesystem "sdb1": xfs_trans_delete_ail: attempting to delete a log item that 
>is not in the AIL
>xfs_force_shutdown(sdb1,0x8) called from line 382 of file 
>fs/xfs/xfs_trans_ail.c.  Return address = 0xc0216a56
>@Linux version 2.6.9 (root@mail1.domain.tld) (gcc version 2.96 20000731 (Red 
>Hat Linux 7.3 2.96-113)) #1 SMP Tue Oct 19 16:04:55 CEST 2004
></SNIP>
>
>
>=======
>Resolution to the mailserver problem:
> 2.4.28 is perfectly stable on these machines.
>
>-------
>Scenario 3: Webservers:
>
>  2.6.10, 2.6.10-ac8 (~3-12 hours uptime):
>
>    <SNIP>
>    Unable to handle kernel paging request
>    <2>SoftDog: Initiating system reboot.
>    <SNIP>
>    (No more...) :(
>
>=======
>Resolution to the webserver problem:
> 2.4.28/2.4.29-rc2 are stable here
>
>-------
>Scenario 4: Storageservers: 
>  2.6.8.1:
>    Oopses after ~5-10 hours whith SMP on. - Cannot find the actual Oopses 
>anymore and 2.6.8+ havent been tested as we cannot afford anymore downtime on 
>these servers.
>
>
>=======
>Resolution to the storage server problem:
> 2.6.8.1 UP is stable (but oopses regularly after memory allocation
> failures)
>
>
>
>Hardware on all servers: IBM x335 and x345.
>
>Mentioned errors seen on a total of 17 servers.
>
>  
>

Re: XFS: inode with st_mode == 0

[Jan Kasprzak]

Christoph Hellwig wrote:
: I have a better patch than the one I gave you (attached below).  If you
: send me a mail with steps to reproduce your remaining problems I'll put
: this very high on my TODO list after christmas.  Btw, any chance you could
: try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9,
: there have been various other fixes in the last months.
: 
	Just FWIW, this patch (applied to 2.6.10) seems to fix the problem
for me. I was not able to reproduce it by running my test script for ~24 hours.

	Thanks!

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Jeffrey E. Hundstad]

Jeffrey Hundstad wrote:

> For more of this look up subjects:
>  Bad things happening to journaled filesystem machines
>  Oops in kjournald
> and from author:
>  Anders Saaby
>
> I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few 
> hours on a machine under real load.   Perhaps us folks with the 
> problem need to talk to the powers who be to come up with a strategy 
> to make a report they can use.  My guess is we're not sending 
> something that can be used.
>
I have found two server in my operation that seem to do quite well on 
linux-2.6.7.  So I believe the brokenness is after this point and before 
linux-2.6.8.1.

...so far I'm not seeing problems after two days with 
linux-2.6.10-ac10.  I'm still crossing my fingers and knocking on wood.

-- 
jeffrey hundstad

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Stephen C. Tweedie]

Hi,

On Mon, 2005-01-17 at 21:31, Jeffrey Hundstad wrote:
> For more of this look up subjects:
>   Bad things happening to journaled filesystem machines
>   Oops in kjournald

That seems to have been due to the xattr problems recently fixed in
Linus's tree.  The xattr race was allowing one process to delete an
unshared xattr block while another was trying to share it, and the
journaling code was getting upset when the second process then tried to
commit the now-deleted block.

--Stephen

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Jeffrey Hundstad]

Stephen C. Tweedie wrote:

>Hi,
>
>On Mon, 2005-01-17 at 21:31, Jeffrey Hundstad wrote:
>  
>
>>For more of this look up subjects:
>>  Bad things happening to journaled filesystem machines
>>  Oops in kjournald
>>    
>>
>
>That seems to have been due to the xattr problems recently fixed in
>Linus's tree.  The xattr race was allowing one process to delete an
>unshared xattr block while another was trying to share it, and the
>journaling code was getting upset when the second process then tried to
>commit the now-deleted block.
>  
>

Thanks for the update.

I wonder if there are several problems.  Alan Cox claimed that there was 
a fix in linux-2.6.10-ac10 that might alleviate the problem.

On linux-2.6.10-ac10 I've got one machine that's been up for 6 days now 
that would never last more then 1 before.  On the other hand I have one 
machine that did die after two days.

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr 
problem fixed?  If so, I'll test there.

-- 
Jeffrey Hundstad

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Stephen C. Tweedie]

Hi,

On Tue, 2005-01-25 at 15:09, Jeffrey Hundstad wrote:

> >>  Bad things happening to journaled filesystem machines
> >>  Oops in kjournald

> I wonder if there are several problems.  Alan Cox claimed that there was 
> a fix in linux-2.6.10-ac10 that might alleviate the problem.

I'm not sure --- there are a couple of bio/bh-related fixes in that
patch, but nothing against jbd/ext3 itself. 

> Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr 
> problem fixed?

Not sure about how much of -ac went in, but it has the xattr fix.

--Stephen

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Jeffrey E. Hundstad]

Stephen C. Tweedie wrote:

>Hi,
>
>On Tue, 2005-01-25 at 15:09, Jeffrey Hundstad wrote:
>
>  
>
>>>> Bad things happening to journaled filesystem machines
>>>> Oops in kjournald
>>>>        
>>>>
>
>  
>
>>I wonder if there are several problems.  Alan Cox claimed that there was 
>>a fix in linux-2.6.10-ac10 that might alleviate the problem.
>>    
>>
>
>I'm not sure --- there are a couple of bio/bh-related fixes in that
>patch, but nothing against jbd/ext3 itself. 
>
>  
>
>>Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr 
>>problem fixed?
>>    
>>
>
>Not sure about how much of -ac went in, but it has the xattr fix.
>
>--Stephen
>
>  
>

I've had my machine that would crash daily if not hourly stay up for 10 
days now.  This is with the linux-2.6.10-ac10 kernel.  I was wondering 
if anyone else is having similiar results.

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Stephen C. Tweedie]

Hi,

On Fri, 2005-01-28 at 20:15, Jeffrey E. Hundstad wrote:

> >>Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr 
> >>problem fixed?

> >Not sure about how much of -ac went in, but it has the xattr fix.

> I've had my machine that would crash daily if not hourly stay up for 10 
> days now.  This is with the linux-2.6.10-ac10 kernel. 

Good to know.  Are you using xattrs extensively (eg. for ACLs, SELinux
or Samba 4)?

--Stephen

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

[Jeffrey E. Hundstad]

Stephen C. Tweedie wrote:

>Hi,
>
>On Fri, 2005-01-28 at 20:15, Jeffrey E. Hundstad wrote:
>
>  
>
>>>>Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr 
>>>>problem fixed?
>>>>        
>>>>
>
>  
>
>>>Not sure about how much of -ac went in, but it has the xattr fix.
>>>      
>>>
>
>  
>
>>I've had my machine that would crash daily if not hourly stay up for 10 
>>days now.  This is with the linux-2.6.10-ac10 kernel. 
>>    
>>
>
>Good to know.  Are you using xattrs extensively (eg. for ACLs, SELinux
>or Samba 4)?
>
>--Stephen
>
>  
>
On the machines that were having problems we really weren't using them 
for anything.  I think I may have been running into the BIO problem that 
was fixed in 2.6.10-ac10.