Corruption problem with ext3 and htree

Corruption problem with ext3 and htree

[Martin Schlemmer]

Hi

For some time now I have been having a problem with ext3 and htree.

I use Gentoo, with portage as package system.  My root is on ext3
without htree, and my portage tmp/build directory is on another
drive with ext3 and htree.

Now, when you install something, it unpacks and compile and then
install it to the build root on the tmp partition (ext3 with htree),
and then 'merge' it to / (ext3 without htree) from that build root.

Every time I get the following when it 'merge' the installed image
to /:

-------------------------------------------------------
>>> /usr/share/man/man3/threads::shared.3pm.gz
>>> /usr/share/man/man3/bigrat.3pm.gz
Traceback (most recent call last):
  File "/usr/bin/emerge", line 1833, in ?
    mydepgraph.merge(mydepgraph.altlist())
  File "/usr/bin/emerge", line 1125, in merge
    retval=portage.doebuild(y,"merge",myroot,edebug)
  File "/usr/lib/python2.2/site-packages/portage.py", line 1467, in
doebuild    return
merge(settings["CATEGORY"],settings["PF"],settings["D"],settings["BUILD
DIR"]+"/build-info",myroot,myebuild=settings["EBUILD"])  File
"/usr/lib/python2.2/site-packages/portage.py", line 1575, in merge   
return mylink.merge(pkgloc,infloc,myroot,myebuild)  File
"/usr/lib/python2.2/site-packages/portage.py", line 4147, in merge   
return self.treewalk(mergeroot,myroot,inforoot,myebuild)  File
"/usr/lib/python2.2/site-packages/portage.py", line 3849, in treewalk   
if
self.mergeme(srcroot,destroot,outfile,secondhand,"",cfgfiledict,mymtime
):  File "/usr/lib/python2.2/site-packages/portage.py", line 4021, in
mergeme    if
self.mergeme(srcroot,destroot,outfile,secondhand,offset+x+"/",cfgfiledi
ct,thismtime):  File "/usr/lib/python2.2/site-packages/portage.py", line
4021, in mergeme    if
self.mergeme(srcroot,destroot,outfile,secondhand,offset+x+"/",cfgfiledi
ct,thismtime):  File "/usr/lib/python2.2/site-packages/portage.py", line
4021, in mergeme    if
self.mergeme(srcroot,destroot,outfile,secondhand,offset+x+"/",cfgfiledi
ct,thismtime):  File "/usr/lib/python2.2/site-packages/portage.py", line
4021, in mergeme    if
self.mergeme(srcroot,destroot,outfile,secondhand,offset+x+"/",cfgfiledi
ct,thismtime):  File "/usr/lib/python2.2/site-packages/portage.py", line
3944, in mergeme    mystat=os.lstat(mysrc)
OSError: [Errno 2] No such file or directory:
'/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::U
til.tmp' 
nosferatu ext3 # rm
/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash\:\:
Util. Hash::Util.3pm.gz  Hash::Util.tmp     
nosferatu ext3 # rm
/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash\:\:
Util.tmp/.deps              Makefile           Makefile.in       
gal-view-menus.h   gal-view-menus.o   .libs              Makefile.am    
   gal-view-menus.c   gal-view-menus.lo  libmenus.la        nosferatu
ext3 #
-------------------------------------------------------

Basically, the .tmp file for the manpage 
(/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::U
til.tmp) gets created as a directory that points to some other dir on
the ext3 with htree partition.

I can recreate this every time.  I can try a few things to try and
figure this out.

Specs:
   P4 2.4 with Asus P4T533-C
   linux-2.5.64 (also with all the previous, and 2.4 ...)


Regards,

-- 

Martin Schlemmer
Gentoo Linux Developer, Desktop/System Team Developer
Cape Town, South Africa

Re: Corruption problem with ext3 and htree

[Andreas Dilger]

On Mar 07, 2003  06:39 +0200, Martin Schlemmer wrote:
> For some time now I have been having a problem with ext3 and htree.
> 
> I use Gentoo, with portage as package system.  My root is on ext3
> without htree, and my portage tmp/build directory is on another
> drive with ext3 and htree.
> 
> Now, when you install something, it unpacks and compile and then
> install it to the build root on the tmp partition (ext3 with htree),
> and then 'merge' it to / (ext3 without htree) from that build root.

There have been a number of ext3+htree fixes in the last week or so.
I'm not sure if all of them are in the kernel yet, but I think the -mm
tree will have the majority of them.  Please also see the ext2-devel
and ext3-users mailing list archives for the last week for the patches.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corruption problem with ext3 and htree

[Martin Schlemmer]

On Fri, 2003-03-07 at 08:48, Andreas Dilger wrote:
> On Mar 07, 2003  06:39 +0200, Martin Schlemmer wrote:
> > For some time now I have been having a problem with ext3 and htree.
> > 
> > I use Gentoo, with portage as package system.  My root is on ext3
> > without htree, and my portage tmp/build directory is on another
> > drive with ext3 and htree.
> > 
> > Now, when you install something, it unpacks and compile and then
> > install it to the build root on the tmp partition (ext3 with htree),
> > and then 'merge' it to / (ext3 without htree) from that build root.
> 
> There have been a number of ext3+htree fixes in the last week or so.
> I'm not sure if all of them are in the kernel yet, but I think the -mm
> tree will have the majority of them.  Please also see the ext2-devel
> and ext3-users mailing list archives for the last week for the patches.
> 

Thanks, will have a try tonight and let you know.


Regards,

-- 
Martin Schlemmer
Gentoo Linux Developer, Desktop Team
Cape Town, South Africa

Re: Corruption problem with ext3 and htree

[Martin Schlemmer]

On Thu, 6 Mar 2003 23:48:19 -0700
Andreas Dilger <adilger@clusterfs.com> wrote:

> There have been a number of ext3+htree fixes in the last week or so.
> I'm not sure if all of them are in the kernel yet, but I think the -mm
> tree will have the majority of them.  Please also see the ext2-devel
> and ext3-users mailing list archives for the last week for the
> patches.
> 

Nope, did not fix it.  I tried to grab all patches from mm2, as well as
any others that was lying around.

------------------------------
man3 # ls Hash\:\:Util.*
ls: Hash::Util.tmp: No such file or directory
Hash::Util.3pm.gz
nosferatu man3 # 
------------------------------

And for some reason its only with the Hash::Util* files that it have
this problem.  Am assuming it might not be related to the filename
itself ?

This is what I get when I fsck it:

--------------------------------------------------------------------
nosferatu root # e2fsck -f /dev/hdg1 
e2fsck 1.32 (09-Nov-2002)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'Hash::Util.tmp' in
/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3 (1619713) has
deleted/unused inode 1619855.  Clear<y>? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/hdg1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/hdg1: 318434/4889248 files (3.0% non-contiguous), 5785602/9771528
blocks
nosferatu root # 
--------------------------------------------------------------------

Like I said, just say how I can try to debug this.

PS: Please CC me, as only subscribed at work.


Regards,

-- 

Martin Schlemmer

Re: Corruption problem with ext3 and htree

[Andreas Dilger]

On Mar 09, 2003  06:33 +0200, Martin Schlemmer wrote:
> Nope, did not fix it.  I tried to grab all patches from mm2, as well as
> any others that was lying around.
> 
> ------------------------------
> man3 # ls Hash\:\:Util.*
> ls: Hash::Util.tmp: No such file or directory
> Hash::Util.3pm.gz
> nosferatu man3 # 
> ------------------------------
> 
> And for some reason its only with the Hash::Util* files that it have
> this problem.  Am assuming it might not be related to the filename
> itself ?
> 
> This is what I get when I fsck it:
> 
> --------------------------------------------------------------------
> nosferatu root # e2fsck -f /dev/hdg1 
> e2fsck 1.32 (09-Nov-2002)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'Hash::Util.tmp' in
> /var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3 (1619713) has
> deleted/unused inode 1619855.  Clear<y>? yes

Out of curiosity, when was the last time before this that the filesystem
was fsck'd?  This sounds a lot like a problem that was (I think) fixed
a couple of months ago relating to renaming files (search for "htree"
in ext2-devel and/or linux-kernel archives around Nov 07, 2002).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corruption problem with ext3 and htree

[Martin Schlemmer]

On Sat, 8 Mar 2003 23:32:19 -0700
Andreas Dilger <adilger@clusterfs.com> wrote:

> Out of curiosity, when was the last time before this that the
> filesystem was fsck'd?  This sounds a lot like a problem that was (I
> think) fixed a couple of months ago relating to renaming files (search
> for "htree" in ext2-devel and/or linux-kernel archives around Nov 07,
> 2002).
> 

Well, several times already since I had this problem.  Cleared out some
stuff, fsck -c 'd it, etc.

If I move the portage's tmpdir to / which is ext3 without dir_index,
then it works just fine.


Regards,

-- 

Martin Schlemmer

Re: Corruption problem with ext3 and htree

[Theodore Ts'o]

Hmm... can you help construct a test case that doesn't rely on the
presence of the Gentoo distribution?  Is there some way we can
instrument the python code so we can see the exact filesystem
operations (renames, deletions, moves, etc.) that is going on?  The
good news is that you say that you're able to reproduce it every
single time, which implies it's not a timing related problem.

> And for some reason its only with the Hash::Util* files that it have
> this problem.  Am assuming it might not be related to the filename
> itself ?

It could possibly be a hash value dependent problem, which case it
could be related to the filename.  That's not very likely, but it is
possible.  If you could send us the result of "dumpe2fs -h /dev/XXXX",
that would be useful.  In particular the last two lines:

  Default directory hash:   tea
  Directory Hash Seed:      407dbbca-8326-4bed-bc7c-bb0453f79049

The most important thing though is to be able to reduce the test case
to something which is slightly easier for us ext2/3 developers to run.  

						- Ted

Re: Corruption problem with ext3 and htree

[Martin Schlemmer]

On Tue, 2003-03-11 at 08:19, Theodore Ts'o wrote:
> Hmm... can you help construct a test case that doesn't rely on the
> presence of the Gentoo distribution?  Is there some way we can
> instrument the python code so we can see the exact filesystem
> operations (renames, deletions, moves, etc.) that is going on?  The
> good news is that you say that you're able to reproduce it every
> single time, which implies it's not a timing related problem.
> 

Ok, will try to get A short way to duplicate this.


Regards,

-- 
Martin Schlemmer

Re: Corruption problem with ext3 and htree

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]

On Tue, 11 Mar 2003 01:19:11 -0500
Theodore Ts'o <tytso@mit.edu> wrote:

> Hmm... can you help construct a test case that doesn't rely on the
> presence of the Gentoo distribution?  Is there some way we can
> instrument the python code so we can see the exact filesystem
> operations (renames, deletions, moves, etc.) that is going on?  The
> good news is that you say that you're able to reproduce it every
> single time, which implies it's not a timing related problem.
> 

I just compile perl-5.8.0, and then install it.

----------------------------------------------------
nosferatu perl-5.8.0 # ls /space/var/tmp/portage/perl-5.8.0-r10/image
nosferatu perl-5.8.0 # make
DESTDIR="/space/var/tmp/portage/perl-5.8.0-r10/image"
INSTALLMAN1DIR="/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/m
an/man1"
INSTALLMAN3DIR="/space/var/tmp/portage/perl-5.8.0-r10/image/foo/usr/sha
re/man/man3" install > /dev/null make[1]: [extras.make] Error 1
(ignored) make[2]: [extras.install] Error 1 (ignored)
nosferatu perl-5.8.0 # ls
/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash\:\:
Util.* -al ls:
/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::Ut
il.tmp: No such file or directory-rw-r--r--    1 root     root        
6435 Mar 11 20:54
/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::Ut
il.3pm nosferatu perl-5.8.0 #
-------------------------------------------------------------

Bad past, thus attached as well.

> It could possibly be a hash value dependent problem, which case it
> could be related to the filename.  That's not very likely, but it is
> possible.  If you could send us the result of "dumpe2fs -h /dev/XXXX",
> that would be useful.  In particular the last two lines:
> 
>   Default directory hash:   tea
>   Directory Hash Seed:      407dbbca-8326-4bed-bc7c-bb0453f79049
> 
> The most important thing though is to be able to reduce the test case
> to something which is slightly easier for us ext2/3 developers to run.
>  

Seems like its creating 'Hash::Util.tmp' as a directory for some reason,
while it should be a 'tmp' file when installing the man pages.

Is there some other hash algorithm I could try ?  Just to verify if it
is that ?  Problem is that If I try to recreate it without the make
install, I do not really succeed.


Regards,

-- 

Martin Schlemmer


[-- Attachment #2: foo --]
[-- Type: application/octet-stream, Size: 795 bytes --]

nosferatu perl-5.8.0 # ls /space/var/tmp/portage/perl-5.8.0-r10/image
nosferatu perl-5.8.0 # make DESTDIR="/space/var/tmp/portage/perl-5.8.0-r10/image" INSTALLMAN1DIR="/space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man1" INSTALLMAN3DIR="/space/var/tmp/portage/perl-5.8.0-r10/image/foo/usr/share/man/man3" install > /dev/null
make[1]: [extras.make] Error 1 (ignored)
make[2]: [extras.install] Error 1 (ignored)
nosferatu perl-5.8.0 # ls /space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash\:\:Util.* -al
ls: /space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::Util.tmp: No such file or directory
-rw-r--r--    1 root     root         6435 Mar 11 20:54 /space/var/tmp/portage/perl-5.8.0-r10/image/usr/share/man/man3/Hash::Util.3pm
nosferatu perl-5.8.0 #