Possible EXT2 File System Corruption in Kernel 2.4

Possible EXT2 File System Corruption in Kernel 2.4

[Frank Krauss]

Good day,

I have a problem in the area of EXT2 File System Corruption.

I attempted originally to send this information to Remy Card as per the
MAINTAINERS file at RemyCard@linux.org but I got the message about
him being an unknown user.

Since my System is fairly up-to-date, both Hardware and Software wise,
I thought that perhaps someone would be able to solve this problem of mine. 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
> Hello Remy,
> 
> I have a problem which appears to be in your area of responsibility and expertise.
> 
> Hardware Information
> 
> 1.  I have a Pentium 4 P.C.
> 2.  The Motherboard is a D850MV.
> 3.  The Chipset is an Intel 850.
> 4.  The System has 256 Mb of Ram.
> 5.  My Hard-drive is a Maxtor 5T030H3 (30 gigibytes)
> 6.  It is on ide1 as /dev/hdc.
> 7.  The disk is Partitioned in the following manner:
>                                 C y l s      Usage              1K Blks
>     A.  /dev/hdc1        1 -   85      Root                660890
>     B.  /dev/hdc2      86 -  150      Swap
>     C.  /dev/hdc3   151 - 3671      /Products      27015180
>     D.  /dev/hdc4 3672 - 3736      /Source            505471
> 
> Software Information
> 
> 1.  My Linux Distribution is Caldera 2.3
> 2.  I have upgraded the Kernel to 2.4.18
> 
> My problem came up yesterday in the following manner:
> My Root Partition, /dev/hdc1 went 100% full.
> Even though I erased various files from it which should have brought
> it down to 97% usage, it remained at the 100% level.
> I believe that this is another, known, separate problem from mine.
> 
> I finally decided to move the /usr/X11R6 directory, which was approx. 134 Mb,
> from the /root partition to a completly empty partition that I had just
> defined as /Products on /dev/hdc3. 
> 
> The Copy worked but I got the following 10 Error Messages:
> 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835885 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835886 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835894 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835902 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835910 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835918 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835926 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835934 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835942 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)): ext2_new_block: Allocating block in system zone - block = 835950 
> 
> Since I did a DU of both the Old and New directories and they were the same,
> I assumed that everything was O.K.
> 
> Unfortunately, after I shutdown the System and re-booted, I got a huge number 
> of INODE error messages that took FSCK about an Hour to fix.
> I did not write any of the Inode information down.
> 
> I see mention on the Net that up to 2.4.17 there are still problems outstanding
> in this area of the Kernel.
> 
> Would you be kind enough to tell me if there is any fix I can put on my
> System to keep this problem from reoccuring again?
> 
> As my System is currently a Testing/Development System, I would be quit happy
> to help you debug this problem with any reasonable testing that you would like
> me to do.
> 
> Alan Cox mentioned, in a response to someone who had a similar problem as
> mine, that a VIA Chipset along with a Sound Blaster Live card, could somehow
> cause this problem.
> 1.   I do NOT have the VIA Chipset on my System.
> 2.   I DO have the Sound Blaster Live card on my System.
> 
> I didn't have this problem using Kernel 2.2.17 on my 486 Computer.
> 
> I would like to get this problem resolved before my System becomes Production
> since I value Reliability and Stability of an Operating System most importantly
> of all.
> 
> I will be glad to provide you any additional information about my System that
> would assist you in solving this problem.
> 
> I would like to thank you in advance for your effort in clearing up this 
> problem effecting the reliability of the New Production Linux Kernel.
> 
> Yours truly,
> 
> Frank Krauss
               

[PATCH] Re: Possible EXT2 File System Corruption in Kernel 2.4

[Andreas Dilger]

On Apr 11, 2002  12:40 -0400, Frank Krauss wrote:
> I have a problem in the area of EXT2 File System Corruption.
> 
> I attempted originally to send this information to Remy Card as per the
> MAINTAINERS file at RemyCard@linux.org but I got the message about
> him being an unknown user.

This is a patch (after I don't know how many years of him not working on
it) to remove Remy Card as ext2 maintainer.  Since I'm not
comfortable adding other people's names as maintainers (and I don't
think Linus/Marcello/Alan would accept that anyways), I've just put
the ext2-devel mailing list.  All of the ext2 developers are on it.

Patch against 2.4.18, but should be fine for 2.4.current, 2.5.current,
but the ext3 part doesn't exist in 2.2.

=============================================================================
--- linux/MAINTAINERS.orig	Fri Dec 21 10:41:53 2001
+++ linux/MAINTAINERS	Thu Apr 11 12:57:13 2002
@@ -544,14 +544,12 @@
 S:      Maintained
 
 EXT2 FILE SYSTEM
-P:	Remy Card
-M:	Remy.Card@linux.org
-L:	linux-kernel@vger.kernel.org
+L:	ext2-devel@lists.sourceforge.net
 S:	Maintained
 
 EXT3 FILE SYSTEM
-P:	Remy Card, Stephen Tweedie
-M:	sct@redhat.com, akpm@zip.com.au, adilger@turbolinux.com
+P:	Stephen Tweedie, Andrew Morton
+M:	sct@redhat.com, akpm@zip.com.au, adilger@clusterfs.com
 L:	ext3-users@redhat.com
 S:	Maintained
 
=============================================================================

> > I got the following 10 Error Messages:
> > 
> > Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)):
> >   ext2_new_block: Allocating block in system zone - block = 835885 
> > Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)):
> >   ext2_new_block: Allocating block in system zone - block = 835886 
> > Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)):
> >   ext2_new_block: Allocating block in system zone - block = 835894 

This is unfortunately only a symptom of a problem and not the original
cause.  There should have previously been errors saying "error - freeing
blocks in system zone".  There is a patch I posted a few months ago to
fix this symptom, but not the actual cause.

Rather than going ahead and allocating these blocks (which is surely a
bad thing, as it will further corrupt the filesystem) the patch marks
the blocks in-use, and continues looking for other blocks to allocate.
Similarly, on the "free" case, it does not mark the blocks as freed in
the bitmap, although it does deallocate it from the inode.  At worst
this will mean that you might have some unusable blocks if the
filesystem is using some feature that the kernel does not understand,
and e2fsck will clean it up for you because it is marked in error.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [PATCH] Re: Possible EXT2 File System Corruption in Kernel 2.4

[Andrew Morton]

Andreas Dilger wrote:
> 
> ...
>  EXT2 FILE SYSTEM
> -P:     Remy Card
> -M:     Remy.Card@linux.org
> -L:     linux-kernel@vger.kernel.org
> +L:     ext2-devel@lists.sourceforge.net
>  S:     Maintained

Yes, I'd support that.   It's a sad thing to do, but there
is no point in misleading people in this way.  Remy already
has an entry in ./CREDITS for ext2.

> This is unfortunately only a symptom of a problem and not the original
> cause.  There should have previously been errors saying "error - freeing
> blocks in system zone".  There is a patch I posted a few months ago to
> fix this symptom, but not the actual cause.

Do you have time to dust that patch off?  Shit happens, and the
filesystems should be robust in the presence of software and
hardware failures.  In this case the filesystem has reached
a point where it *knows* that it is about to do the wrong
thing, it emits a warning and then it just goes ahead and does it!

-

Re: [PATCH] Possible EXT2 File System Corruption in Kernel 2.4

[Andreas Dilger]

On Apr 11, 2002  12:40 -0400, Frank Krauss wrote:
> I have a problem in the area of EXT2 File System Corruption.
> 
> My problem came up yesterday in the following manner:
> My Root Partition, /dev/hdc1 went 100% full.
> Even though I erased various files from it which should have brought
> it down to 97% usage, it remained at the 100% level.
> I believe that this is another, known, separate problem from mine.
> 
> The Copy worked but I got the following 10 Error Messages:
> 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)):
>    ext2_new_block: Allocating block in system zone - block = 835885 
> Apr  9 17:12:51 kernel: EXT2-fs error (device ide1(22,3)):
>    ext2_new_block: Allocating block in system zone - block = 835886 

OK, here is my updated patch for fixing the SYMPTOM of this problem and
not the root CAUSE of the problem.  The symptom is that ext2/ext3 will
happily free and allocate blocks that are supposed to be filesystem
metadata if there is an error in the block bitmaps.  Errors could
previously crop up in the block bitmaps if, say, an inode had corrupt
block numbers, or an indirect block had garbage in it.

The patch changes the code so that it still complains about trying to
allocate or free such a block, but then it "does the right thing" by
keeping the block allocated for (what it thinks is) filesystem metadata.
If, in some future revision of ext2, we change the location of some
metadata, at worst we will have a few blocks which cannot be allocated
by an older kernel, but e2fsck will clean up at a later time.  That is
far preferrable to corrupting the superblock or bitmaps or inode table,
which will, in turn, lead to more filesystem corruption.

The patch has been in my kernel tree for a long time, but is relatively
untested as it only handles error situations.  It seems pretty obvious,
however, so it should go into the kernel proper.  Patch is against 2.4.18,
but does not intersect with Andrew's recent fixes to filesystem-full
handling in ext3_new_block() in 2.4.19-pre7.  It also includes the checks
for block free overflows that have been in 2.5-dj for a good while.

Cheers, Andreas
====================== ext2-2.4.18-badalloc.diff =====================
diff -ru linux-2.4.18.orig/fs/ext2/balloc.c linux-2.4.18-aed/fs/ext2/balloc.c
--- linux-2.4.18.orig/fs/ext2/balloc.c	Wed Feb 27 10:31:58 2002
+++ linux-2.4.18-aed/fs/ext2/balloc.c	Mon Mar 18 17:07:55 2002
@@ -269,7 +269,8 @@
 	}
 	lock_super (sb);
 	es = sb->u.ext2_sb.s_es;
-	if (block < le32_to_cpu(es->s_first_data_block) || 
+	if (block < le32_to_cpu(es->s_first_data_block) ||
+	    block + count < block ||
 	    (block + count) > le32_to_cpu(es->s_blocks_count)) {
 		ext2_error (sb, "ext2_free_blocks",
 			    "Freeing blocks not in datazone - "
@@ -302,22 +303,20 @@
 	if (!gdp)
 		goto error_return;
 
-	if (in_range (le32_to_cpu(gdp->bg_block_bitmap), block, count) ||
-	    in_range (le32_to_cpu(gdp->bg_inode_bitmap), block, count) ||
-	    in_range (block, le32_to_cpu(gdp->bg_inode_table),
-		      sb->u.ext2_sb.s_itb_per_group) ||
-	    in_range (block + count - 1, le32_to_cpu(gdp->bg_inode_table),
-		      sb->u.ext2_sb.s_itb_per_group))
-		ext2_error (sb, "ext2_free_blocks",
-			    "Freeing blocks in system zones - "
-			    "Block = %lu, count = %lu",
-			    block, count);
+	for (i = 0; i < count; i++, block++) {
+		if (block == le32_to_cpu(gdp->bg_block_bitmap) ||
+		    block == le32_to_cpu(gdp->bg_inode_bitmap) ||
+		    in_range(block, le32_to_cpu(gdp->bg_inode_table),
+			     EXT2_SB(sb)->s_itb_per_group)) {
+			ext2_error(sb, __FUNCTION__,
+				   "Freeing block in system zone - block = %lu",
+				   block);
+			continue;
+		}
 
-	for (i = 0; i < count; i++) {
 		if (!ext2_clear_bit (bit + i, bh->b_data))
-			ext2_error (sb, "ext2_free_blocks",
-				      "bit already cleared for block %lu",
-				      block + i);
+			ext2_error(sb, __FUNCTION__,
+				   "bit already cleared for block %lu", block);
 		else {
 			DQUOT_FREE_BLOCK(inode, 1);
 			gdp->bg_free_blocks_count =
@@ -336,7 +335,6 @@
 		wait_on_buffer (bh);
 	}
 	if (overflow) {
-		block += count;
 		count = overflow;
 		goto do_more;
 	}
@@ -522,8 +520,12 @@
 	    in_range (tmp, le32_to_cpu(gdp->bg_inode_table),
 		      sb->u.ext2_sb.s_itb_per_group))
 		ext2_error (sb, "ext2_new_block",
-			    "Allocating block in system zone - "
-			    "block = %u", tmp);
+			    "Allocating block in system zone - block = %lu",
+			    tmp);
+		ext2_set_bit(j, bh->b_data);
+		DQUOT_FREE_BLOCK(inode, 1);
+		goto repeat;
+	}
 
 	if (ext2_set_bit (j, bh->b_data)) {
 		ext2_warning (sb, "ext2_new_block",
--- linux-2.4.18.orig/fs/ext3/balloc.c	Wed Feb 27 10:31:59 2002
+++ linux-2.4.18-aed/fs/ext3/balloc.c	Mon Mar 18 17:15:46 2002
@@ -276,7 +273,8 @@
 	}
 	lock_super (sb);
 	es = sb->u.ext3_sb.s_es;
-	if (block < le32_to_cpu(es->s_first_data_block) || 
+	if (block < le32_to_cpu(es->s_first_data_block) ||
+	    block + count < block ||
 	    (block + count) > le32_to_cpu(es->s_blocks_count)) {
 		ext3_error (sb, "ext3_free_blocks",
 			    "Freeing blocks not in datazone - "
@@ -309,17 +307,6 @@
 	if (!gdp)
 		goto error_return;
 
-	if (in_range (le32_to_cpu(gdp->bg_block_bitmap), block, count) ||
-	    in_range (le32_to_cpu(gdp->bg_inode_bitmap), block, count) ||
-	    in_range (block, le32_to_cpu(gdp->bg_inode_table),
-		      sb->u.ext3_sb.s_itb_per_group) ||
-	    in_range (block + count - 1, le32_to_cpu(gdp->bg_inode_table),
-		      sb->u.ext3_sb.s_itb_per_group))
-		ext3_error (sb, "ext3_free_blocks",
-			    "Freeing blocks in system zones - "
-			    "Block = %lu, count = %lu",
-			    block, count);
-
 	/*
 	 * We are about to start releasing blocks in the bitmap,
 	 * so we need undo access.
@@ -345,14 +332,24 @@
 	if (err)
 		goto error_return;
 
-	for (i = 0; i < count; i++) {
+	for (i = 0; i < count; i++, block++) {
+		if (block == le32_to_cpu(gdp->bg_block_bitmap) ||
+		    block == le32_to_cpu(gdp->bg_inode_bitmap) ||
+		    in_range(block, le32_to_cpu(gdp->bg_inode_table),
+			     sb->u.ext2_sb.s_itb_per_group)) {
+			ext3_error(sb, __FUNCTION__,
+				   "Freeing block in system zone - block = %lu",
+				   block);
+			continue;
+		}
+
 		/*
 		 * An HJ special.  This is expensive...
 		 */
 #ifdef CONFIG_JBD_DEBUG
 		{
 			struct buffer_head *debug_bh;
-			debug_bh = sb_get_hash_table(sb, block + i);
+			debug_bh = sb_get_hash_table(sb, block);
 			if (debug_bh) {
 				BUFFER_TRACE(debug_bh, "Deleted!");
 				if (!bh2jh(bitmap_bh)->b_committed_data)
@@ -365,9 +362,8 @@
 #endif
 		BUFFER_TRACE(bitmap_bh, "clear bit");
 		if (!ext3_clear_bit (bit + i, bitmap_bh->b_data)) {
-			ext3_error (sb, __FUNCTION__,
-				      "bit already cleared for block %lu", 
-				      block + i);
+			ext3_error(sb, __FUNCTION__,
+				   "bit already cleared for block %lu", block);
 			BUFFER_TRACE(bitmap_bh, "bit already cleared");
 		} else {
 			dquot_freed_blocks++;
@@ -415,7 +411,6 @@
 	if (!err) err = ret;
 
 	if (overflow && !err) {
-		block += count;
 		count = overflow;
 		goto do_more;
 	}
@@ -575,6 +574,7 @@
 
 	ext3_debug ("goal=%lu.\n", goal);
 
+repeat:
 	/*
 	 * First, test whether the goal block is free.
 	 */
@@ -684,10 +684,21 @@
 	if (tmp == le32_to_cpu(gdp->bg_block_bitmap) ||
 	    tmp == le32_to_cpu(gdp->bg_inode_bitmap) ||
 	    in_range (tmp, le32_to_cpu(gdp->bg_inode_table),
-		      sb->u.ext3_sb.s_itb_per_group))
-		ext3_error (sb, "ext3_new_block",
-			    "Allocating block in system zone - "
-			    "block = %u", tmp);
+		      EXT3_SB(sb)->s_itb_per_group)) {
+		ext3_error(sb, __FUNCTION__,
+			   "Allocating block in system zone - block = %u", tmp);
+
+		/* Note: This will potentially use up one of the handle's
+		 * buffer credits.  Normally we have way too many credits,
+		 * so that is OK.  In _very_ rare cases it might not be OK.
+		 * We will trigger an assertion if we run out of credits,
+		 * and we will have to do a full fsck of the filesystem -
+		 * better than randomly corrupting filesystem metadata.
+		 */
+		ext3_set_bit(j, bh->b_data);
+		goto repeat;
+	}
+
 
 	/* The superblock lock should guard against anybody else beating
 	 * us to this point! */
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [PATCH] Possible EXT2 File System Corruption in Kernel 2.4

[Juan Quintela]

>>>>> "andreas" == Andreas Dilger <adilger@clusterfs.com> writes:

Hi

Sorry for reading so late that mail.

andreas> diff -ru linux-2.4.18.orig/fs/ext2/balloc.c linux-2.4.18-aed/fs/ext2/balloc.c
andreas> --- linux-2.4.18.orig/fs/ext2/balloc.c	Wed Feb 27 10:31:58 2002
andreas> +++ linux-2.4.18-aed/fs/ext2/balloc.c	Mon Mar 18 17:07:55 2002
andreas> @@ -269,7 +269,8 @@
andreas> }
andreas> lock_super (sb);
andreas> es = sb->u.ext2_sb.s_es;
andreas> -	if (block < le32_to_cpu(es->s_first_data_block) || 
andreas> +	if (block < le32_to_cpu(es->s_first_data_block) ||
andreas> +	    block + count < block ||

It is just me, or this will allways be false?  A fast grep shows that
count is always bigger than 1. Same for the ext3 part.

Later, Juan.

-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: [PATCH] Possible EXT2 File System Corruption in Kernel 2.4

[Andreas Dilger]

On May 20, 2002  11:38 +0200, Juan Quintela wrote:
> >>>>> "andreas" == Andreas Dilger <adilger@clusterfs.com> writes:
> andreas> diff -ru linux-2.4.18.orig/fs/ext2/balloc.c linux-2.4.18-aed/fs/ext2/balloc.c
> andreas> --- linux-2.4.18.orig/fs/ext2/balloc.c	Wed Feb 27 10:31:58 2002
> andreas> +++ linux-2.4.18-aed/fs/ext2/balloc.c	Mon Mar 18 17:07:55 2002
> andreas> @@ -269,7 +269,8 @@
> andreas> }
> andreas> lock_super (sb);
> andreas> es = sb->u.ext2_sb.s_es;
> andreas> -	if (block < le32_to_cpu(es->s_first_data_block) || 
> andreas> +	if (block < le32_to_cpu(es->s_first_data_block) ||
> andreas> +	    block + count < block ||
> 
> It is just me, or this will allways be false?  A fast grep shows that
> count is always bigger than 1. Same for the ext3 part.

Well, under normal operation this test will never be true (same as the
other tests there), but it appears that in some cases there _are_ values
of "block + count" which pass this test but fail later on.  I don't know
the exact source of the problem, but it appears to happen when "block"
is -1, and "block + count" is then 0, which does not trigger the
existing tests (s_first_data_block is 0 for 4kB filesystems).

It is likely that block is being set as -EPERM or something like that,
but I'm not sure.  In any case, you can check l-k archives for the
original emails on this thread where the users report trying to access
group 131071, which is 2^32 / 32768 (i.e. -1 / number of blocks per
group for a 4kB block ext2 filesystem).

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [PATCH] Possible EXT2 File System Corruption in Kernel 2.4

[David S. Miller]

   From: Andreas Dilger <adilger@clusterfs.com>
   Date: Mon, 20 May 2002 21:57:02 -0600

   It is likely that block is being set as -EPERM or something like that,
   but I'm not sure.

Interesting analysis.

However, I walked over this code a few times and I cannot
find a way that -EPERM can land there.

What might be happening, instead, is that due to some bug
in ext2_alloc_block we end up with -1 as the answer.  It
would be useful to add some debugging there to see if the
return value 'j' is ever -1 when we set *err to '0'.