* Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 @ 2015-04-15 15:18 Akos Csete 2015-04-15 19:06 ` Darrick J. Wong 0 siblings, 1 reply; 6+ messages in thread From: Akos Csete @ 2015-04-15 15:18 UTC (permalink / raw) To: linux-ext4 I ran into a very long execution time, approaching 48 hours, with e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks. I've been tailing the e2fsck output combined with a timestamp and found progress is extremely slow. Is this slow processing expected? I checked Ted's response to a 2009 posting for ext3 with a similar issue where he recommended selectively wiping inodes with debugfs clri then restoring files as needed. Are there any other options for ext4, perhaps other/faster repair alternatives? Thanks, Akos ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 2015-04-15 15:18 Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 Akos Csete @ 2015-04-15 19:06 ` Darrick J. Wong 2015-04-16 15:00 ` Theodore Ts'o 0 siblings, 1 reply; 6+ messages in thread From: Darrick J. Wong @ 2015-04-15 19:06 UTC (permalink / raw) To: Akos Csete; +Cc: linux-ext4 On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote: > I ran into a very long execution time, approaching 48 hours, with > e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after > fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks. > I've been tailing the e2fsck output combined with a timestamp and > found progress is extremely slow. > Is this slow processing expected? I checked Ted's response to a 2009 > posting for ext3 with a similar issue where he recommended selectively > wiping inodes with debugfs clri then restoring files as needed. Are > there any other options for ext4, perhaps other/faster repair > alternatives? Unfortunately, the multiple-blocks cleaner runs one block at a time and is quite slow. 1.42.12 might improve things somewhat since I cleaned out some of the stranger things it would do, but it's still not fast. (Some day in the future maybe we'll support reflink in which case this whole part of fsck can go away.) --D > > Thanks, > Akos > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 2015-04-15 19:06 ` Darrick J. Wong @ 2015-04-16 15:00 ` Theodore Ts'o 2015-04-17 20:05 ` Akos Csete 0 siblings, 1 reply; 6+ messages in thread From: Theodore Ts'o @ 2015-04-16 15:00 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Akos Csete, linux-ext4 On Wed, Apr 15, 2015 at 12:06:14PM -0700, Darrick J. Wong wrote: > On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote: > > I ran into a very long execution time, approaching 48 hours, with > > e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after > > fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks. > > I've been tailing the e2fsck output combined with a timestamp and > > found progress is extremely slow. > > Is this slow processing expected? I checked Ted's response to a 2009 > > posting for ext3 with a similar issue where he recommended selectively > > wiping inodes with debugfs clri then restoring files as needed. Are > > there any other options for ext4, perhaps other/faster repair > > alternatives? > > Unfortunately, the multiple-blocks cleaner runs one block at a time and is > quite slow. 1.42.12 might improve things somewhat since I cleaned out some of > the stranger things it would do, but it's still not fast. > > (Some day in the future maybe we'll support reflink in which case this whole > part of fsck can go away.) Actually, reflink won't make the problem go away. What probably *will* make this problem much less of an issue is when the metadata checksum code is fully supported with the relesae of e2fsprogs 1.43, since the most common cause of the problem is when the storage device screws up and stores blocks to the wrong location on disk. One way of recovering from this is to take the output to date in the pass 1b/c/d output, and to examine the inodes which are multiply claimed. Either the inodes will look like total garbage, or more likely, there will be a contiguous sequence of 16 or 32 inodes which look identical to a contiguous sequence of 16 or 32 inodes somewhere else. So the trick is to use debugfs and figure out which contiguous set of inodes appear to be correct, by correlating the file names of those inodes (assuming the directory can be found) with the inode contents and/or types and/or user/gorupownership. Once you figure out which set of 16 or 32 inodes are the bad copy, delete them using debugfs's clri command. The pass1b/c/d algorithsm are O(n**2), and since a chunk of inodes can be resolved/eliminated at a time, after you clear up a chunk of the inodes, you can rerun e2fsck, and hopefully the next run will take less time. If it still stalls out, you can take the output of e2fsck and use it to clear out more inode numbers. - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 2015-04-16 15:00 ` Theodore Ts'o @ 2015-04-17 20:05 ` Akos Csete 2015-04-17 21:08 ` Theodore Ts'o 0 siblings, 1 reply; 6+ messages in thread From: Akos Csete @ 2015-04-17 20:05 UTC (permalink / raw) To: Theodore Ts'o; +Cc: Darrick J. Wong, linux-ext4 Thanks for the helpful suggestions. So manually reducing the number of multiply-claimed blocks would save significant time for e2fsck's phase of cloning blocks in such cases. I assume that running fsck.ext4 -nf between repairs and summing the number of multiply-claimed blocks reported in pass 1d would be indicative of the remaining work, although only half of the total needs to be cloned/deleted. In the particular scenario I was looking at a storage side error started blocking writes from the machine. Per e2fsck the resulting contiguous block allocations for new files were claiming subsets of contiguous blocks belonging to older files. For e.g. inode #172163416 below was claiming blocks from older files: ... Pass 1B: Rescanning for multiply-claimed blocks ... Multiply-claimed block(s) in inode 161484221: => lists 2,465 multiply-claimed contiguous blocks Multiply-claimed block(s) in inode 161484230: => lists 51,687 multiply-claimed blocks, 4 subsets of contiguous blocks Multiply-claimed block(s) in inode 161484232: => lists 4,096 multiply-claimed contiguous blocks Multiply-claimed block(s) in inode 161484266: => lists 6,503 multiply-claimed contiguous blocks Multiply-claimed block(s) in inode 161494510: => lists 24,576 multiply-claimed contiguous blocks Multiply-claimed block(s) in inode 172163416: => lists 86,792 multiply-claimed blocks, 2 subsets of contiguous blocks ... Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks ... File /archive/DNS-000.00.9.253_201503311401_queries.log (inode #161484221, mod time Tue Mar 31 15:05:32 2015) has 2465 multiply-claimed block(s), shared with 1 file(s): /import/201504091101_queries.log (inode #172163416, mod time Thu Apr 9 18:07:22 2015) ... File /archive/DNS-000.00.6.253_201503311401_queries.log (inode #161484230, mod time Tue Mar 31 15:05:26 2015) has 51687 multiply-claimed block(s), shared with 143 file(s): ... File /archive/DNS-000.00.10.252_201503311401_queries.log (inode #161484232, mod time Tue Mar 31 15:05:31 2015) has 4096 multiply-claimed block(s), shared with 1 file(s): /import/201504091101_queries.log (inode #172163416, mod time Thu Apr 9 18:07:22 2015) ... File /archive/DNS-000.00.15.252_201503311401_queries.log (inode #161484266, mod time Tue Mar 31 15:05:23 2015) has 6503 multiply-claimed block(s), shared with 1 file(s): /import/201504091101_queries.log (inode #172163416, mod time Thu Apr 9 18:07:22 2015) ... File /archive/DNS-000.00.6.253_201504030701_queries.log (inode #161494510, mod time Fri Apr 3 08:05:12 2015) has 24576 multiply-claimed block(s), shared with 1 file(s): /import/201504091101_queries.log (inode #172163416, mod time Thu Apr 9 18:07:22 2015) ... File /import/201504091101_queries.log (inode #172163416, mod time Thu Apr 9 18:07:22 2015) has 86792 multiply-claimed block(s), shared with 5 file(s): /archive/DNS-000.00.9.253_201503311401_queries.log (inode #161484221, mod time Tue Mar 31 15:05:32 2015) /archive/DNS-000.00.15.252_201503311401_queries.log (inode #161484266, mod time Tue Mar 31 15:05:23 2015) /archive/DNS-000.00.10.252_201503311401_queries.log (inode #161484232, mod time Tue Mar 31 15:05:31 2015) /archive/DNS-000.00.6.253_201503311401_queries.log (inode #161484230, mod time Tue Mar 31 15:05:26 2015) /archive/DNS-000.00.6.253_201504030701_queries.log (inode #161494510, mod time Fri Apr 3 08:05:12 2015) ... Could I then just wipe the inodes of the older files (and restore them later from backup) and let the new file claim those blocks? Or are the new file's claim to those blocks suspect? Thanks, Akos On Thu, Apr 16, 2015 at 11:00 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Wed, Apr 15, 2015 at 12:06:14PM -0700, Darrick J. Wong wrote: >> On Wed, Apr 15, 2015 at 11:18:18AM -0400, Akos Csete wrote: >> > I ran into a very long execution time, approaching 48 hours, with >> > e2fsck 1.42.9 when running e2fsck -fyv on a 7.8TB volume after >> > fsck.ext4 -nf found 1761 inodes containing multiply-claimed blocks. >> > I've been tailing the e2fsck output combined with a timestamp and >> > found progress is extremely slow. >> > Is this slow processing expected? I checked Ted's response to a 2009 >> > posting for ext3 with a similar issue where he recommended selectively >> > wiping inodes with debugfs clri then restoring files as needed. Are >> > there any other options for ext4, perhaps other/faster repair >> > alternatives? >> >> Unfortunately, the multiple-blocks cleaner runs one block at a time and is >> quite slow. 1.42.12 might improve things somewhat since I cleaned out some of >> the stranger things it would do, but it's still not fast. >> >> (Some day in the future maybe we'll support reflink in which case this whole >> part of fsck can go away.) > > Actually, reflink won't make the problem go away. What probably > *will* make this problem much less of an issue is when the metadata > checksum code is fully supported with the relesae of e2fsprogs 1.43, > since the most common cause of the problem is when the storage device > screws up and stores blocks to the wrong location on disk. > > One way of recovering from this is to take the output to date in the > pass 1b/c/d output, and to examine the inodes which are multiply > claimed. Either the inodes will look like total garbage, or more > likely, there will be a contiguous sequence of 16 or 32 inodes which > look identical to a contiguous sequence of 16 or 32 inodes somewhere > else. So the trick is to use debugfs and figure out which contiguous > set of inodes appear to be correct, by correlating the file names of > those inodes (assuming the directory can be found) with the inode > contents and/or types and/or user/gorupownership. Once you figure out > which set of 16 or 32 inodes are the bad copy, delete them using > debugfs's clri command. > > The pass1b/c/d algorithsm are O(n**2), and since a chunk of inodes can > be resolved/eliminated at a time, after you clear up a chunk of the > inodes, you can rerun e2fsck, and hopefully the next run will take > less time. If it still stalls out, you can take the output of e2fsck > and use it to clear out more inode numbers. > > - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 2015-04-17 20:05 ` Akos Csete @ 2015-04-17 21:08 ` Theodore Ts'o 2015-04-18 15:02 ` Andreas Dilger 0 siblings, 1 reply; 6+ messages in thread From: Theodore Ts'o @ 2015-04-17 21:08 UTC (permalink / raw) To: Akos Csete; +Cc: Darrick J. Wong, linux-ext4 On Fri, Apr 17, 2015 at 04:05:04PM -0400, Akos Csete wrote: > > Could I then just wipe the inodes of the older files (and restore them > later from backup) and let the new file claim those blocks? Or are the > new file's claim to those blocks suspect? OK, if the mod times are different, then it probably wasn't a case of an inode getting written to the wrong place on disk. What probably did happen is the block allocation bitmaps got corrupted, and so blocks that were in use (and should have been marked in use) were not, so they got used for some other inode. So yes, you can try wiping the inodes for the older files, and hope for the best, but it may be that some of the newer files may end up getting corrupted. So you might want to do some spot checks afterwards. And if you have backups, this might be a good time to consider going to your backup tapes.... - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 2015-04-17 21:08 ` Theodore Ts'o @ 2015-04-18 15:02 ` Andreas Dilger 0 siblings, 0 replies; 6+ messages in thread From: Andreas Dilger @ 2015-04-18 15:02 UTC (permalink / raw) To: Theodore Ts'o; +Cc: Akos Csete, Darrick J. Wong, linux-ext4 I had multiple-linked block problems this week as a CF card I was using in a PVR started failing. The problem was just garbage in the inside table that happened to have a bit of a pattern and caused many duplicate blocks. One of the patches we have for Lustre, which unfortunately wasn't installed on this system, is the ability to prefer erasing inodes with duplicate blocks instead of cloning them. I think the last time I tried to push this patch it got hung up because I don't understand how to make the e2fsck.conf and command line defaults interact properly. http://git.hpdd.intel.com/tools/e2fsprogs.git/patch/1f9018d5ddf56d5feb9ef156c9684aee6d79cdee The other patch that would be very useful in this case is "inode badness" which tries to detect if inodes are garbage and just delete them instead of turning them into something that appears good in small increments. http://git.hpdd.intel.com/tools/e2fsprogs.git/patch/8528b0480b132dc9fae37a2e2efe5e9c24c3d56f Ted didn't like the approach this patch had taken, because he thought it touched e2fsck in too many places. I was thinking of a different way of doing this, by hooking into fix_problem() if pctx->ino is set, but I'm not yet sure if that will work. Cheers, Andreas >> On Apr 17, 2015, at 15:08, Theodore Ts'o <tytso@mit.edu> wrote: >> >> On Fri, Apr 17, 2015 at 04:05:04PM -0400, Akos Csete wrote: >> >> Could I then just wipe the inodes of the older files (and restore them >> later from backup) and let the new file claim those blocks? Or are the >> new file's claim to those blocks suspect? > > OK, if the mod times are different, then it probably wasn't a case of > an inode getting written to the wrong place on disk. What probably > did happen is the block allocation bitmaps got corrupted, and so > blocks that were in use (and should have been marked in use) were not, > so they got used for some other inode. > > So yes, you can try wiping the inodes for the older files, and hope > for the best, but it may be that some of the newer files may end up > getting corrupted. So you might want to do some spot checks > afterwards. And if you have backups, this might be a good time to > consider going to your backup tapes.... > > - Ted > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-04-18 15:02 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-04-15 15:18 Fsck repair takes very long for 1,761 inodes containing multiply-claimed blocks on ext4 Akos Csete 2015-04-15 19:06 ` Darrick J. Wong 2015-04-16 15:00 ` Theodore Ts'o 2015-04-17 20:05 ` Akos Csete 2015-04-17 21:08 ` Theodore Ts'o 2015-04-18 15:02 ` Andreas Dilger
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.