Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Akos Csete]

I ran into a very long execution time, approaching 48 hours, with
e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after
fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks.
I've been tailing the e2fsck output combined with a timestamp and
found progress is extremely slow.
Is this slow processing expected? I checked Ted's response to a 2009
posting for ext3 with a similar issue where he recommended selectively
wiping inodes with debugfs clri then restoring files as needed. Are
there any other options for ext4, perhaps other/faster repair
alternatives?

Thanks,
Akos

Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Darrick J. Wong]

On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote:
> I ran into a very long execution time, approaching 48 hours, with
> e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after
> fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks.
> I've been tailing the e2fsck output combined with a timestamp and
> found progress is extremely slow.
> Is this slow processing expected? I checked Ted's response to a 2009
> posting for ext3 with a similar issue where he recommended selectively
> wiping inodes with debugfs clri then restoring files as needed. Are
> there any other options for ext4, perhaps other/faster repair
> alternatives?

Unfortunately, the multiple-blocks cleaner runs one block at a time and is
quite slow.  1.42.12 might improve things somewhat since I cleaned out some of
the stranger things it would do, but it's still not fast.

(Some day in the future maybe we'll support reflink in which case this whole
part of fsck can go away.)

--D

> 
> Thanks,
> Akos
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Theodore Ts'o]

On Wed, Apr 15, 2015 at 12:06:14PM -0700, Darrick J. Wong wrote:
> On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote:
> > I ran into a very long execution time, approaching 48 hours, with
> > e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after
> > fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks.
> > I've been tailing the e2fsck output combined with a timestamp and
> > found progress is extremely slow.
> > Is this slow processing expected? I checked Ted's response to a 2009
> > posting for ext3 with a similar issue where he recommended selectively
> > wiping inodes with debugfs clri then restoring files as needed. Are
> > there any other options for ext4, perhaps other/faster repair
> > alternatives?
> 
> Unfortunately, the multiple-blocks cleaner runs one block at a time and is
> quite slow.  1.42.12 might improve things somewhat since I cleaned out some of
> the stranger things it would do, but it's still not fast.
> 
> (Some day in the future maybe we'll support reflink in which case this whole
> part of fsck can go away.)

Actually, reflink won't make the problem go away.  What probably
*will* make this problem much less of an issue is when the metadata
checksum code is fully supported with the relesae of e2fsprogs 1.43,
since the most common cause of the problem is when the storage device
screws up and stores blocks to the wrong location on disk.

One way of recovering from this is to take the output to date in the
pass 1b/c/d output, and to examine the inodes which are multiply
claimed. Either the inodes will look like total garbage, or more
likely, there will be a contiguous sequence of 16 or 32 inodes which
look identical to a contiguous sequence of 16 or 32 inodes somewhere
else.  So the trick is to use debugfs and figure out which contiguous
set of inodes appear to be correct, by correlating the file names of
those inodes (assuming the directory can be found) with the inode
contents and/or types and/or user/gorupownership.  Once you figure out
which set of 16 or 32 inodes are the bad copy, delete them using
debugfs's clri command.

The pass1b/c/d algorithsm are O(n**2), and since a chunk of inodes can
be resolved/eliminated at a time, after you clear up a chunk of the
inodes, you can rerun e2fsck, and hopefully the next run will take
less time.  If it still stalls out, you can take the output of e2fsck
and use it to clear out more inode numbers.

						- Ted

Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Akos Csete]

Thanks for the helpful suggestions. So manually reducing the number of
multiply-claimed blocks would save significant time for e2fsck's phase
of cloning blocks in such cases. I assume that running fsck.ext4 -nf
between repairs and summing the number of multiply-claimed blocks
reported in pass 1d would be indicative of the remaining work,
although only half of the total needs to be cloned/deleted.

In the particular scenario I was looking at a storage side error
started blocking writes from the machine. Per e2fsck the resulting
contiguous block allocations for new files were claiming subsets of
contiguous blocks belonging to older files. For e.g. inode #172163416
below was claiming blocks from older files:

...
Pass 1B: Rescanning for multiply-claimed blocks
...
Multiply-claimed block(s) in inode 161484221: => lists 2,465
multiply-claimed contiguous blocks
Multiply-claimed block(s) in inode 161484230: => lists 51,687
multiply-claimed blocks, 4 subsets of contiguous blocks
Multiply-claimed block(s) in inode 161484232: => lists 4,096
multiply-claimed contiguous blocks
Multiply-claimed block(s) in inode 161484266: => lists 6,503
multiply-claimed contiguous blocks
Multiply-claimed block(s) in inode 161494510: => lists 24,576
multiply-claimed contiguous blocks
Multiply-claimed block(s) in inode 172163416: => lists 86,792
multiply-claimed blocks, 2 subsets of contiguous blocks
...
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
...
File /archive/DNS-000.00.9.253_201503311401_queries.log (inode
#161484221, mod time Tue Mar 31 15:05:32 2015)
  has 2465 multiply-claimed block(s), shared with 1 file(s):
/import/201504091101_queries.log (inode #172163416, mod time Thu Apr
9 18:07:22 2015)
...
File /archive/DNS-000.00.6.253_201503311401_queries.log (inode
#161484230, mod time Tue Mar 31 15:05:26 2015)
  has 51687 multiply-claimed block(s), shared with 143 file(s):
...
File /archive/DNS-000.00.10.252_201503311401_queries.log (inode
#161484232, mod time Tue Mar 31 15:05:31 2015)
  has 4096 multiply-claimed block(s), shared with 1 file(s):
/import/201504091101_queries.log (inode #172163416, mod time Thu Apr
9 18:07:22 2015)
...
File /archive/DNS-000.00.15.252_201503311401_queries.log (inode
#161484266, mod time Tue Mar 31 15:05:23 2015)
  has 6503 multiply-claimed block(s), shared with 1 file(s):
/import/201504091101_queries.log (inode #172163416, mod time Thu Apr
9 18:07:22 2015)
...
File /archive/DNS-000.00.6.253_201504030701_queries.log (inode
#161494510, mod time Fri Apr  3 08:05:12 2015)
  has 24576 multiply-claimed block(s), shared with 1 file(s):
  /import/201504091101_queries.log (inode #172163416, mod time Thu Apr
 9 18:07:22 2015)
...
File /import/201504091101_queries.log (inode #172163416, mod time Thu
Apr  9 18:07:22 2015)
  has 86792 multiply-claimed block(s), shared with 5 file(s):
/archive/DNS-000.00.9.253_201503311401_queries.log (inode #161484221,
mod time Tue Mar 31 15:05:32 2015)
/archive/DNS-000.00.15.252_201503311401_queries.log (inode #161484266,
mod time Tue Mar 31 15:05:23 2015)
/archive/DNS-000.00.10.252_201503311401_queries.log (inode #161484232,
mod time Tue Mar 31 15:05:31 2015)
/archive/DNS-000.00.6.253_201503311401_queries.log (inode #161484230,
mod time Tue Mar 31 15:05:26 2015)
/archive/DNS-000.00.6.253_201504030701_queries.log (inode #161494510,
mod time Fri Apr  3 08:05:12 2015)
...

Could I then just wipe the inodes of the older files (and restore them
later from backup) and let the new file claim those blocks? Or are the
new file's claim to those blocks suspect?

Thanks,
Akos

On Thu, Apr 16, 2015 at 11:00 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Wed, Apr 15, 2015 at 12:06:14PM -0700, Darrick J. Wong wrote:
>> On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote:
>> > I ran into a very long execution time, approaching 48 hours, with
>> > e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after
>> > fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks.
>> > I've been tailing the e2fsck output combined with a timestamp and
>> > found progress is extremely slow.
>> > Is this slow processing expected? I checked Ted's response to a 2009
>> > posting for ext3 with a similar issue where he recommended selectively
>> > wiping inodes with debugfs clri then restoring files as needed. Are
>> > there any other options for ext4, perhaps other/faster repair
>> > alternatives?
>>
>> Unfortunately, the multiple-blocks cleaner runs one block at a time and is
>> quite slow.  1.42.12 might improve things somewhat since I cleaned out some of
>> the stranger things it would do, but it's still not fast.
>>
>> (Some day in the future maybe we'll support reflink in which case this whole
>> part of fsck can go away.)
>
> Actually, reflink won't make the problem go away.  What probably
> *will* make this problem much less of an issue is when the metadata
> checksum code is fully supported with the relesae of e2fsprogs 1.43,
> since the most common cause of the problem is when the storage device
> screws up and stores blocks to the wrong location on disk.
>
> One way of recovering from this is to take the output to date in the
> pass 1b/c/d output, and to examine the inodes which are multiply
> claimed. Either the inodes will look like total garbage, or more
> likely, there will be a contiguous sequence of 16 or 32 inodes which
> look identical to a contiguous sequence of 16 or 32 inodes somewhere
> else.  So the trick is to use debugfs and figure out which contiguous
> set of inodes appear to be correct, by correlating the file names of
> those inodes (assuming the directory can be found) with the inode
> contents and/or types and/or user/gorupownership.  Once you figure out
> which set of 16 or 32 inodes are the bad copy, delete them using
> debugfs's clri command.
>
> The pass1b/c/d algorithsm are O(n**2), and since a chunk of inodes can
> be resolved/eliminated at a time, after you clear up a chunk of the
> inodes, you can rerun e2fsck, and hopefully the next run will take
> less time.  If it still stalls out, you can take the output of e2fsck
> and use it to clear out more inode numbers.
>
>                                                 - Ted

Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Theodore Ts'o]

On Fri, Apr 17, 2015 at 04:05:04PM -0400, Akos Csete wrote:
> 
> Could I then just wipe the inodes of the older files (and restore them
> later from backup) and let the new file claim those blocks? Or are the
> new file's claim to those blocks suspect?

OK, if the mod times are different, then it probably wasn't a case of
an inode getting written to the wrong place on disk.  What probably
did happen is the block allocation bitmaps got corrupted, and so
blocks that were in use (and should have been marked in use) were not,
so they got used for some other inode.

So yes, you can try wiping the inodes for the older files, and hope
for the best, but it may be that some of the newer files may end up
getting corrupted.  So you might want to do some spot checks
afterwards.  And if you have backups, this might be a good time to
consider going to your backup tapes....

						- Ted

Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4

[Andreas Dilger]

I had multiple-linked block problems this week as a CF card I was using in
a PVR started failing. The problem was just garbage in the inside table
that happened to have a bit of a pattern and caused many duplicate
blocks. 

One of the patches we have for Lustre, which unfortunately wasn't
installed on this system, is the ability to prefer erasing inodes with
duplicate blocks instead of cloning them. I think the last time I tried to
push this patch it got hung up because I don't understand how to
make the e2fsck.conf and command line defaults interact properly. 

http://git.hpdd.intel.com/tools/e2fsprogs.git/patch/1f9018d5ddf56d5feb9ef156c9684aee6d79cdee

The other patch that would be very useful in this case is "inode badness"
which tries to detect if inodes are garbage and just delete them instead
of turning them into something that appears good in small increments.

http://git.hpdd.intel.com/tools/e2fsprogs.git/patch/8528b0480b132dc9fae37a2e2efe5e9c24c3d56f

Ted didn't like the approach this patch had taken, because he thought
it touched e2fsck in too many places. I was thinking of a different way of
doing this, by hooking into fix_problem() if pctx->ino is set, but
I'm not yet sure if that will work. 

Cheers, Andreas

>> On Apr 17, 2015, at 15:08, Theodore Ts'o <tytso@mit.edu> wrote:
>> 
>> On Fri, Apr 17, 2015 at 04:05:04PM -0400, Akos Csete wrote:
>> 
>> Could I then just wipe the inodes of the older files (and restore them
>> later from backup) and let the new file claim those blocks? Or are the
>> new file's claim to those blocks suspect?
> 
> OK, if the mod times are different, then it probably wasn't a case of
> an inode getting written to the wrong place on disk.  What probably
> did happen is the block allocation bitmaps got corrupted, and so
> blocks that were in use (and should have been marked in use) were not,
> so they got used for some other inode.
> 
> So yes, you can try wiping the inodes for the older files, and hope
> for the best, but it may be that some of the newer files may end up
> getting corrupted.  So you might want to do some spot checks
> afterwards.  And if you have backups, this might be a good time to
> consider going to your backup tapes....
> 
>                        - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html