Re: [Automated-testing] syzkaller reproducers

[George Kennedy <george.kennedy@oracle.com>]

On 5/16/2022 3:41 AM, Dmitry Vyukov wrote:
> On Fri, 13 May 2022 at 14:16, Dmitry Vyukov <dvyukov@google.com> wrote:
>> On Fri, 13 May 2022 at 13:35, George Kennedy <george.kennedy@oracle.com> wrote:
>>>>> + Shuah, George and mailing lists
>>>>>
>>>>> Google groups did not CC all of the members in the thread for my
>>>>> reply. It would be valuable to get everyone's feedback on this thread.
>>>>>
>>>>> On Sat, May 7, 2022 at 11:34 PM Palash <oswalpalash@gmail.com> wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> Reviving this old thread. Do we think there's value in automatically publishing a C reproducer(if it exists) whenever syzkaller determines a reported bug is patched and fixed?
>>>>>> This allows us to track these reproducers more diligently and we only have reproducers for the bugs that have been patched.
>>>>>> If this is valuable, then we could think about answering these questions -
>>>>>> 1. Is there a particular way we want to format the reproducers? (Example : SKIP/OK/FAIL/TIMEOUT (if output is possible))
>>>>>> 2. Where do we want to submit the reproducers to be added to the testing project and what is the process we want to use?
>>>>>> 3. Is it valuable to record the crash report information for the original fuzzer crash when the c reproducer is filed to add context to it; in case developers want to troubleshoot? This could save valuable debugging time.
>>>>>>
>>>>>> Happy to contribute on this change to the upstream fuzzing system.
>>>> More automation is definitely welcome.
>>>> I am not sure how widely used is the current base of reproducers. But
>>>> George keeps asking for updates and I keep failing at doing it in a
>>>> timely manner.
>>> We run the ~6000 reproducers as a regression test. Test takes ~4 hours.
>>>
>>> Having the reproducer before there is a patch is valuable to us as we
>>> know whether or not our distro branches have the same failure.
>>>
>>> As a way to reduce test time, some sort of repro classification would be
>>> helpful (i.e. open/closed/invalid).
>>>
>>> For crash troubleshooting info we try to match the crash string with
>>> syzbot's crash string.
>>>
>>> Thank you,
>>> George
>>>> It would be easiest for the syzbot app to upload reproducers to a GCS
>>>> bucket. But I am not sure what's a good format for this. Everything I
>>>> can think of is somewhat clumsy. It could upload weekly archives, but
>>>> then there is an issue of discoverability of new archives (since GCS
>>>> is HTTP rather than FTP).
>>>>
>>>> Not sure if it's feasible to give it an auth token for a github repo
>>>> to push individual files as we do now.
>>>>
>>>> Re format. Bonus points if the system will be able to regenerate and
>>>> update meta-information. The chances that we will come up with the
>>>> ultimately right format from day 1 are low. But that may complicate
>>>> the implementation.
>> The most feasible for syzbot automation seems to be to push to a
>> public GCS bucket and then users will need to do "gsutil rsync" to
>> download updates.
>>
>> Or another option is to expose a new json API to get reproducers list
>> and then to fetch individual reproducers. Though this will need some
>> custom tooling on the user side to fetch updates.
>>
>> Re exporting status open/closed/invalid and e.g. fixing patch, that
>> would also be more natural to expose via the API since this
>> information changes dynamically.
>>
>> It's also possible to use both. Namely, reproducers are uploaded to
>> the GCS bucket and each repro file contains just some bug ID. Then the
>> bug ID can be used to fetch any additional dynamically changing info
>> via json API.
>
> I've pushed a new drop of 898 reproducers:
> https://github.com/dvyukov/syzkaller-repros/commit/026c8891a19e672bba71481be250080e1a302ed8

Thank you,
George