Re: [PATCH] erofs: move erofs out of staging

[Gao Xiang <hsiangkao@aol.com>]

Hi Darrick,

On Mon, Aug 19, 2019 at 09:09:23AM -0700, Darrick J. Wong wrote:
> On Mon, Aug 19, 2019 at 04:14:11AM +0800, Gao Xiang wrote:
> > Hi all,
> > 
> > On Mon, Aug 19, 2019 at 02:16:55AM +0800, Gao Xiang wrote:
> > > Hi Hch,
> > > 
> > > On Sun, Aug 18, 2019 at 10:47:02AM -0700, Christoph Hellwig wrote:
> > > > On Sun, Aug 18, 2019 at 10:29:38AM -0700, Eric Biggers wrote:
> > > > > Not sure what you're even disagreeing with, as I *do* expect new filesystems to
> > > > > be held to a high standard, and to be written with the assumption that the
> > > > > on-disk data may be corrupted or malicious.  We just can't expect the bar to be
> > > > > so high (e.g. no bugs) that it's never been attained by *any* filesystem even
> > > > > after years/decades of active development.  If the developers were careful, the
> > > > > code generally looks robust, and they are willing to address such bugs as they
> > > > > are found, realistically that's as good as we can expect to get...
> > > >
> > > > Well, the impression I got from Richards quick look and the reply to it is
> > > > that there is very little attempt to validate the ondisk data structure
> > > > and there is absolutely no priority to do so.  Which is very different
> > > > from there is a bug or two here and there.
> > > 
> > > As my second reply to Richard, I didn't fuzz all the on-disk fields for EROFS.
> > > and as my reply to Richard / Greg, current EROFS is used on the top of dm-verity.
> > > 
> > > I cannot say how well EROFS will be performed on malformed images (and you can
> > > also find the bug richard pointed out is a miswritten break->continue by myself).
> > > 
> > > I posted the upstream EROFS post on July 4, 2019 and a month and a half later,
> > > no one can tell me (yes, thanks for kind people reply me about their suggestion)
> > > what we should do next (you can see these emails, I sent many times) to meet
> > > the minimal upstream requirements and rare people can even dip into my code.
> > > 
> > > That is all I want to say. I will work on autofuzz these days, and I want to
> > > know how to meet your requirements on this (you can tell us your standard,
> > > how well should we do).
> > > 
> > > OK, you don't reply to my post once, I have no idea how to get your first reply.
> > 
> > I have made a simple fuzzer to inject messy in inode metadata,
> > dir data, compressed indexes and super block,
> > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?h=experimental-fuzzer
> > 
> > I am testing with some given dirs and the following script.
> > Does it look reasonable?
> > 
> > # !/bin/bash
> > 
> > mkdir -p mntdir
> > 
> > for ((i=0; i<1000; ++i)); do
> > 	mkfs/mkfs.erofs -F$i testdir_fsl.fuzz.img testdir_fsl > /dev/null 2>&1
> 
> mkfs fuzzes the image? Er....

Thanks for your reply.

First, This is just the first step of erofs fuzzer I wrote yesterday night...

> 
> Over in XFS land we have an xfs debugging tool (xfs_db) that knows how
> to dump (and write!) most every field of every metadata type.  This
> makes it fairly easy to write systematic level 0 fuzzing tests that
> check how well the filesystem reacts to garbage data (zeroing,
> randomizing, oneing, adding and subtracting small integers) in a field.
> (It also knows how to trash entire blocks.)

Actually, compared with XFS, EROFS has rather simple on-disk format.
What we inject one time is quite deterministic.

The first step just purposely writes some random fuzzed data to
the base inode metadata, compressed indexes, or dir data field
(one round one field) to make it validity and coverability.

> 
> You might want to write such a debugging tool for erofs so that you can
> take apart crashed images to get a better idea of what went wrong, and
> to write easy fuzzing tests.

Yes, we will do such a debugging tool of course. Actually Li Guifu is now
developping a erofs-fuse to support old linux versions or other OSes for
archiveing only use, we will base on that code to develop a better fuzzer
tool as well.

Thanks,
Gao Xiang

> 
> --D
> 
> > 	umount mntdir
> > 	mount -t erofs -o loop testdir_fsl.fuzz.img mntdir
> > 	for j in `find mntdir -type f`; do
> > 		md5sum $j > /dev/null
> > 	done
> > done
> > 
> > Thanks,
> > Gao Xiang
> > 
> > > 
> > > Thanks,
> > > Gao Xiang
> > >