Re: [LSF/MM TOPIC] online filesystem repair

["Darrick J. Wong" <darrick.wong@oracle.com>]

On Sun, Jan 15, 2017 at 04:01:30PM -0800, Viacheslav Dubeyko wrote:
> On Fri, 2017-01-13 at 23:54 -0800, Darrick J. Wong wrote:
> > Hi,
> > 
> > I've been working on implementing online metadata scrubbing and
> > repair
> > in XFS.��Most of the code is self contained inside XFS, but there's a
> > small amount of interaction with the VFS freezer code that has to
> > happen
> > in order to shut down the filesystem to rebuild the extent backref
> > records.��It might be interesting to discuss the (fairly slight)
> > requirements upon the VFS to support repairs, and/or have a BoF to
> > discuss how to build an online checker if any of the other
> > filesystems
> > are interested in this.
> > 
> 
> How do you imagine a generic way to support repairs for different file
> systems? From one point of view, to have generic way of the online file
> system repairing could be the really great subsystem.

I don't, sadly.  There's not even a way to /check/ all fs metadata in a
"generic" manner -- we can use the standard VFS interfaces to read
all metadata, but this is fraught.  Even if we assume the fs can spot
check obviously garbage values, that's still not the appropriate place
for a full scan.

> But, from another point of view, every file system has own
> architecture, own set of metadata and own way to do fsck
> check/recovering.

Yes, and this wouldn't change.  The particular mechanism of fixing a
piece of metadata will always be fs-dependent, but the thing that I'm
interested in discussing is how do we avoid having these kinds of things
interact badly with the VFS?

> As far as I can judge, there are significant amount of research
> efforts in this direction (Recon [1], [2], for example).

Yes, I remember Recon.  I appreciated the insight that while it's
impossible to block everything for a full scan, it /is/ possible to
check a single object and its relation to other metadata items.  The xfs
scrubber also takes an incremental approach to verifying a filesystem;
we'll lock each metadata object and verify that its relationships with
the other metadata make sense.  So long as we aren't bombarding the fs
with heavy metadata update workloads, of course.

On the repair side of things xfs added reverse-mapping records, which
the repair code uses to regenerate damaged primary metadata.  After we
land inode parent pointers we'll be able to do the same reconstructions
that we can now do for block allocations...

...but there are some sticky problems with repairing the reverse
mappings.  The normal locking order for that part of xfs is sb_writers
-> inode -> ag header -> rmap btree blocks, but to repair we have to
freeze the filesystem against writes so that we can scan all the inodes.

> But we still haven't any real general online file system repair
> subsystem in the Linux kernel.

I think the ocfs2 developers have encoded some ability to repair
metadata over the past year, though it seems limited to fixing some
parts of inodes.  btrfs stores duplicate copies and restores when
necessary, I think.  Unfortunately, fixing disk corruption is something
that's not easily genericized, which means that I don't think we'll ever
achieve a general subsystem.

But we could at least figure out what in the VFS has to change (if
anything) to support this type of usage.

> Do you have some new insight? What's difference of your
> vision? If we have online file system repair subsystem then how file
> system driver will need to interact with the goal to make internal
> repairing?

It's pretty much all private xfs userspace ioctls[1] with a driver
program[2].

--D

[1] https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=djwong-devel
[2] https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=djwong-devel

> 
> Thanks,
> Vyacheslav Dubeyko.
> 
> [1]�http://www.eecg.toronto.edu/~ashvin/publications/recon-fs-consistency-runtime.pdf
> [2]�https://www.researchgate.net/publication/269300836_Managing_the_file_system_from_the_kernel
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html