From: Lukas Bulwahn <email@example.com> To: Thorsten Leemhuis <firstname.lastname@example.org>, Paul Albertella <email@example.com> Cc: Dmitry Vyukov <firstname.lastname@example.org>, email@example.com, LKML <firstname.lastname@example.org>, Greg Kroah-Hartman <email@example.com>, Guillaume Tucker <firstname.lastname@example.org>, email@example.com, Sasha Levin <firstname.lastname@example.org>, Marco Elver <email@example.com>, syzkaller <firstname.lastname@example.org>, Mara Mihali <email@example.com> Subject: Re: finding regressions with syzkaller Date: Wed, 22 Sep 2021 13:21:18 +0200 [thread overview] Message-ID: <CAKXUXMxg-eywEYrR0oSAo14F7CmiYAT7VDxV71U4-Tv8E0LeVQ@mail.gmail.com> (raw) In-Reply-To: <firstname.lastname@example.org> On Wed, Aug 11, 2021 at 1:25 PM Thorsten Leemhuis <email@example.com> wrote: > > [CCing Lukas] > > Hi Dmitry! > > On 10.08.21 19:08, Dmitry Vyukov wrote: > > [...] > > The idea is to generate random test programs (as syzkaller does) and > > then execute them on 2 different kernels and compare results (so > > called "differential fuzzing"). This has the potential of finding not > > just various "crashes" but also logical bugs and regressions. > > Hmmm, interesting concept! > > > The major issue is various false positive differences caused by > > timings, non-determinism, accumulated state, intentional and > > semi-intentional changes (e.g. subtle API extensions), etc. We learnt > > how to deal with some of these to some degree, but feasibility is > > still an open question. > > Sounds complicated and like a lot of manual work. > > Do you have in mind that Linus and hence many other Kernel developers > afaics only care about regressions someone actually observed in a > practice? Like a software or script breaking due to a kernel-side change? > > To quote Linus from > https://lore.kernel.org/lkml/CA+55aFx3RswnjmCErk8QhCo0KrCvxZnuES3WALBR1NkPbUZ8qw@mail.gmail.com/ > > ```The Linux "no regressions" rule is not about some theoretical > "the ABI changed". It's about actual observed regressions. > > So if we can improve the ABI without any user program or workflow > breaking, that's fine.``` > > His stance on that afaik has not changed since then. > > Thus after ruling our all false positives syzkaller might find, there > will always be the follow-up question "well, does anything/anyone > actually care?". That might be hard to answer and requires yet more > manual work by some human. Maybe this working hours at least for now are > better spend in other areas. > > > Since this work is in very early stage, I only have very high-level questions: > > - what do you think about feasibility/usefulness of this idea in general? > > TBH I'm a bit sceptical due to the above factors. Don't get me wrong, > making syzkaller look out for regressions sounds great, but I wonder if > there are more pressing issues that are worth getting at first. > > Another aspect: CI testing already finds quite a few regressions, but > those that are harder to catch are afaics often in driver code. And you > often can't test that without the hardware, which makes me assume that > syzkaller wouldn't help here (or am I wrong?) > > > - any suggestions on how to make the tool find more differences/bugs > > or how to make it more reliable? > > - is there a list or pointers to some known past regressions that > > would be useful to find with such tool? (I've looked at the things > > reported on the regressions@ list, but it's mostly crashes/not > > booting, but that's what syzkaller can find already well) > > I first wanted to tell you "look up the reports I compiled in 2017 in > the LKML archives", but I guess the way better solution is: just grep > for "regression" in the commit log. > > > - anybody else we should CC? > > I guess the people from the Elisa project might be interested in this, > that's why I CCed Lukas. > Thanks, Thorsten. I do follow the syzkaller mailing list, so I have seen that email before, but I do appreciate your implicit acknowledgement here :) ... and Dmitry is back from vacation and I guess we will hear more today at the Testing and Fuzzing MC on this topic. Further people/lists to CC are: Paul Albertella <firstname.lastname@example.org> (already CCed here) I am personally certainly interested and I think this work gives companies in the area of building trustable software and systems (see Paul's area of expertise) a good understanding how reliable and to which extent the statement "all Linux kernels are backwards compatible" really holds. I unfortunately lost the Fuzzing Team (Jouni Högander, Jukka Kaartinen et al.) previously working with me, and I need to first get back some budget, build up a new team and hope that we can then also follow this idea and contribute here as well. (Fingers crossed that I can convince some others to give me money and work with me on this...) Looking forward to the presentation at the MC. Best regards, Lukas
prev parent reply other threads:[~2021-09-22 11:21 UTC|newest] Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-08-10 17:08 Dmitry Vyukov 2021-08-11 11:25 ` Thorsten Leemhuis 2021-08-12 9:15 ` Dmitry Vyukov 2021-09-22 11:21 ` Lukas Bulwahn [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAKXUXMxg-eywEYrR0oSAo14F7CmiYAT7VDxV71U4-Tv8E0LeVQ@mail.gmail.com \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: finding regressions with syzkaller' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).