Re: Running syzkaller repros using kvm-xfstests

From: "Theodore Y. Ts'o" <tytso@mit.edu>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: Eric Biggers <ebiggers@google.com>,
	Dave Chinner <david@fromorbit.com>,
	Matthew Wilcox <willy@infradead.org>,
	linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	syzkaller <syzkaller@googlegroups.com>,
	Al Viro <viro@zeniv.linux.org.uk>
Subject: Re: Running syzkaller repros using kvm-xfstests
Date: Sun, 8 Apr 2018 14:02:52 -0400	[thread overview]
Message-ID: <20180408180252.GC9720@thunk.org> (raw)
In-Reply-To: <CACT4Y+Ynbn9qAmk3jVBs6pUjDAz6gJMD5YHM4+EyzEuwBj+4-Q@mail.gmail.com>

On Sun, Apr 08, 2018 at 03:18:39PM +0200, Dmitry Vyukov wrote:
> 
> But note that syzkaller is under active development, so pre-canned
> binaries may not always work. Mismatching binary may not understand
> all syscalls, fail to parse program, interpret arguments differently,
> execute program differently, setup a different environment for the
> test, etc. Now a C program captures all of this, because code that
> transforms syzkaller programs into C is versioned along with the rest
> of the system.
> Strictly saying, for syzkaller reproducers one needs to use the exact
> syzkaller revision listed along with the reproducer, see for example:
> https://syzkaller.appspot.com/bug?id=3fb9c4777053e79a6d2a65ac3738664c87629a21
> The "#syz test" styzbot command does this. Using a different syzkaller
> revision may or may not work.

Thanks for the warning.  I assume you try to maintain backwards
compatibility where possible?  It might be nice if you could add some
kind of explicit versioning scheme --- perhaps with a major/minor
version scheme where the syz-executor needs to have the same major
number, and a minor number >= the minor version number of the test?

One of the reasons why the C program is not so useful for me is that
in the Repeat:true case, the C program repeats forever.  So for
example, I translate Repeat:true to -repeat=100.  See:

https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/usr/local/bin/run-syz

I suppose I could just abort the test after N minutes and assume if
the kernel hasn't crashed, that it's probably not going to.  But some
way that the C program can be given an argument or an environment
variable to control how number of loops it will run might be useful.
And some kind of hint as how reliable the repro would be (e.g,. some
indication that you should try to run it at least N times to get a
failure at least 95% of the time).

						- Ted