Re: Running syzkaller repros using kvm-xfstests

[Dmitry Vyukov <dvyukov@google.com>]

On Sun, Apr 8, 2018 at 8:02 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Sun, Apr 08, 2018 at 03:18:39PM +0200, Dmitry Vyukov wrote:
>>
>> But note that syzkaller is under active development, so pre-canned
>> binaries may not always work. Mismatching binary may not understand
>> all syscalls, fail to parse program, interpret arguments differently,
>> execute program differently, setup a different environment for the
>> test, etc. Now a C program captures all of this, because code that
>> transforms syzkaller programs into C is versioned along with the rest
>> of the system.
>> Strictly saying, for syzkaller reproducers one needs to use the exact
>> syzkaller revision listed along with the reproducer, see for example:
>> https://syzkaller.appspot.com/bug?id=3fb9c4777053e79a6d2a65ac3738664c87629a21
>> The "#syz test" styzbot command does this. Using a different syzkaller
>> revision may or may not work.
>
> Thanks for the warning.  I assume you try to maintain backwards
> compatibility where possible?  It might be nice if you could add some
> kind of explicit versioning scheme --- perhaps with a major/minor
> version scheme where the syz-executor needs to have the same major
> number, and a minor number >= the minor version number of the test?

We try to not break backwards compatibility without a reason.

Preserving full backwards compatibility within a single binary is
extremely hard. It's like asking kernel to support each and every ever
existed version of every in-memory data structure and all of the
non-functional aspects (like any fluctuations in performance). If one
could give us several additional FTEs for this, then it might be
doable. But even then I don't  think it's the best use of the FTE time
because version control system already gives us exactly this -- exact
behavior on a past revision. On top of this, the backward
compatibility support will sure have bugs too. In the best case we
will spent time debugging why a new version does not precisely model
behavior of an old version. In the worst case you will test something
and think that the bug is fixed, but it's just that the new version
does not behave exactly as the old one. On top of this, this still
does not give us forward compatibility, something that one wants in
majority of cases with an old pre-canned binary. On top of this, the
binaries will be huge because they will need to capture exact versions
of all system call descriptions (and the simplest option for this is
keeping copies all versions), a 87 MiB image definitely won't be
enough to hold this, the binary will be somewhere between gigs and
tens of gigs.


> One of the reasons why the C program is not so useful for me is that
> in the Repeat:true case, the C program repeats forever.  So for
> example, I translate Repeat:true to -repeat=100.  See:
>
> https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/usr/local/bin/run-syz
>
> I suppose I could just abort the test after N minutes and assume if
> the kernel hasn't crashed, that it's probably not going to.  But some
> way that the C program can be given an argument or an environment
> variable to control how number of loops it will run might be useful.
> And some kind of hint as how reliable the repro would be (e.g,. some
> indication that you should try to run it at least N times to get a
> failure at least 95% of the time).

I think:

timeout -s KILL 450 ./a.out

is the solution.
Repro logic runs programs for at most 7.5 minutes, so 450 should be good.

Re env var. There are opposite views too. People complain that
syzkaller C repros are mess (which they are). Currently they complain
minimal amount of code to reproduce the bugs. If we also start
staffing some aux logic in them, it won't be helpful. timeout command
looks just as good.