Re: fuzzing bcachefs with dm-flakey

[Mikulas Patocka <mpatocka@redhat.com>]

On Mon, 29 May 2023, Kent Overstreet wrote:

> On Mon, May 29, 2023 at 04:59:40PM -0400, Mikulas Patocka wrote:
> > Hi
> > 
> > I improved the dm-flakey device mapper target, so that it can do random 
> > corruption of read and write bios - I uploaded it here: 
> > https://people.redhat.com/~mpatocka/testcases/bcachefs/dm-flakey.c
> > 
> > I set up dm-flakey, so that it corrupts 10% of read bios and 10% of write 
> > bios with this command:
> > dmsetup create flakey --table "0 `blockdev --getsize /dev/ram0` flakey /dev/ram0 0 0 1 4 random_write_corrupt 100000000 random_read_corrupt 100000000"
> 
> I've got some existing ktest tests for error injection:
> https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/single_device.ktest#n200
> https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/replication.ktest#n491
> 
> I haven't looked at dm-flakey before, I take it you're silently
> corrupting data instead of just failing the IOs like these tests do?

Yes, silently corrupting.

When I tried to simulate I/O errors with dm-flakey, bcachefs worked 
correcly - there were no errors returned to userspace and no crashes.

Perhaps, it should treat metadata checksum errors in the same way as disk 
failures?

> Let's add what you're doing to ktest, and see if we can merge it with
> the existing tests.

> Good catches on all of them. Darrick's been on me to get fuzz testing
> going, looks like it's definitely needed :)
> 
> However, there's two things I want in place first before I put much
> effort into fuzz testing:
> 
>  - Code coverage analysis. ktest used to have integrated code coverage
>    analysis, where you'd tell it a subdirectory of the kernel tree
>    (doing code coverage analysis for the entire kernel is impossibly
>    slow) and it would run tests and then give you the lcov output.
> 
>    However, several years ago something about kbuild changed, and the
>    method ktest was using for passing in build flags for a specific
>    subdir on the command line stopped working. I would like to track
>    down someone who understands kbuild and get this working again.
> 
>  - Fault injection
> 
>    Years and years ago, when I was still at Google and this was just
>    bcache, we had fault injection that worked like dynamic debug: you
>    could call dynamic_fault("type of fault") anywhere in your code,
>    and it returned a bool indicating whether that fault had been enabled
>    - and faults were controllable at runtime via debugfs, we had tests
>    that iterated over e.g. faults in the initialization path, or memory
>    allocation failures, and flipped them on one by one and ran
>    $test_workload.
> 
>    The memory allocation profiling stuff that Suren and I have been
>    working on includes code tagging, which is for (among other things) a
>    new and simplified implementation of dynamic fault injection, which
>    I'm going to push forward again once the memory allocation profiling
>    stuff gets merged.
> 
> The reason I want this stuff is because fuzz testing tends to be a
> heavyweight, scattershot approach.
> 
> I want to be able to look at the code coverage analysis first to e.g.
> work on a chunk of code at a time and make sure it's tested thoroughly,
> instead of jumping around in the code at random depending on what fuzz
> testing finds, and when we are fuzz testing I want to be able to add
> fault injection points and write unit tests so that we can have much
> more targeted, quicker to run tests going forward.
> 
> Can I get you interested in either of those things? I'd really love to
> find someone to hand off or collaborate with on the fault injection
> stuff in particular.

I'd like to know how do you want to do coverage analysis? By instrumenting 
each branch and creating a test case that tests that the branch goes both 
ways?

I know that people who write spacecraft-grade software do such tests, but 
I can't quite imagine how would that work in a filesystem.

"grep -w if fs/bcachefs/*.[ch] | wc -l" shows that there are 5828 
conditions. That's one condition for every 15.5 lines.

Mikulas