From mboxrd@z Thu Jan 1 00:00:00 1970 MIME-Version: 1.0 References: <876a2abe-41ab-5819-4ae8-ad26186d0d1c@kernel.org> <226099bc-9763-3a73-e26a-b292f601494c@kernel.org> <20191011180248.GA24089@rei.lan> <20191014085414.GB31760@rei.lan> <62903a33-8ffc-56b6-de1a-539f10b5de2a@oracle.com> <86bde120-e5fe-4bb1-9b93-769a444500f9@oracle.com> <8a4dbbb1-f8ba-00ba-41d2-d82a35fc0f81@oracle.com> <1cecad9d-36cf-4703-3f2e-6b2454761f0f@oracle.com> In-Reply-To: <1cecad9d-36cf-4703-3f2e-6b2454761f0f@oracle.com> From: "Dmitry Vyukov" Date: Fri, 13 May 2022 14:16:16 +0200 Message-ID: Subject: Re: [Automated-testing] syzkaller reproducers Content-Type: text/plain; charset="UTF-8" List-ID: To: George Kennedy Cc: kernelci@groups.io, oswalpalash@gmail.com, syzkaller , Shuah Khan , automated-testing@yoctoproject.org, Vegard Nossum On Fri, 13 May 2022 at 13:35, George Kennedy wrote: > >> + Shuah, George and mailing lists > >> > >> Google groups did not CC all of the members in the thread for my > >> reply. It would be valuable to get everyone's feedback on this thread. > >> > >> On Sat, May 7, 2022 at 11:34 PM Palash wrote: > >>> Hello all, > >>> > >>> Reviving this old thread. Do we think there's value in automatically publishing a C reproducer(if it exists) whenever syzkaller determines a reported bug is patched and fixed? > >>> This allows us to track these reproducers more diligently and we only have reproducers for the bugs that have been patched. > >>> If this is valuable, then we could think about answering these questions - > >>> 1. Is there a particular way we want to format the reproducers? (Example : SKIP/OK/FAIL/TIMEOUT (if output is possible)) > >>> 2. Where do we want to submit the reproducers to be added to the testing project and what is the process we want to use? > >>> 3. Is it valuable to record the crash report information for the original fuzzer crash when the c reproducer is filed to add context to it; in case developers want to troubleshoot? This could save valuable debugging time. > >>> > >>> Happy to contribute on this change to the upstream fuzzing system. > > > > More automation is definitely welcome. > > I am not sure how widely used is the current base of reproducers. But > > George keeps asking for updates and I keep failing at doing it in a > > timely manner. > We run the ~6000 reproducers as a regression test. Test takes ~4 hours. > > Having the reproducer before there is a patch is valuable to us as we > know whether or not our distro branches have the same failure. > > As a way to reduce test time, some sort of repro classification would be > helpful (i.e. open/closed/invalid). > > For crash troubleshooting info we try to match the crash string with > syzbot's crash string. > > Thank you, > George > > > > It would be easiest for the syzbot app to upload reproducers to a GCS > > bucket. But I am not sure what's a good format for this. Everything I > > can think of is somewhat clumsy. It could upload weekly archives, but > > then there is an issue of discoverability of new archives (since GCS > > is HTTP rather than FTP). > > > > Not sure if it's feasible to give it an auth token for a github repo > > to push individual files as we do now. > > > > Re format. Bonus points if the system will be able to regenerate and > > update meta-information. The chances that we will come up with the > > ultimately right format from day 1 are low. But that may complicate > > the implementation. The most feasible for syzbot automation seems to be to push to a public GCS bucket and then users will need to do "gsutil rsync" to download updates. Or another option is to expose a new json API to get reproducers list and then to fetch individual reproducers. Though this will need some custom tooling on the user side to fetch updates. Re exporting status open/closed/invalid and e.g. fixing patch, that would also be more natural to expose via the API since this information changes dynamically. It's also possible to use both. Namely, reproducers are uploaded to the GCS bucket and each repro file contains just some bug ID. Then the bug ID can be used to fetch any additional dynamically changing info via json API. > >>> Thanks, > >>> Palash > >>> On Friday, March 13, 2020 at 11:59:31 AM UTC-4 shuah wrote: > >>>> On 3/3/20 1:45 AM, Dmitry Vyukov wrote: > >>>>> Shauh, > >>>>> > >>>>> We've added more reproducers to: > >>>>> https://github.com/dvyukov/syzkaller-repros/tree/master/linux > >>>>> > >>>>> It makes sense to pull in them to the kernel-arts repo. Not sure > >>>>> what's the most convenient way for you since it's not exactly a > >>>>> traditional "patch"? Perhaps you just copy linux/*.c files and commit? > >>>>> > >>>> Thanks Dmitry. I will pull these in. I will pull these in. > >>>> > >>>> thanks, > >>>> -- Shuah